Combination of AWE (Criterion) Feedback with the Process Approach and Its Impact on EFL Writing Content/Idea Development and Organization

While being very important and useful, in certain English as a foreign language (EFL) setting the act of providing feedback on writing constructs considered to be time-consuming that can lack meaning and focus, especially if not done well. In this regard, the development and use of automated writing evaluation (AWE) software is anticipated to assist EFL learners to improve writing skills through feedback as well as helping teachers by saving time and effort and supporting their writing instruction. This paper examines the influence of automated feedback (through the use of Criterion) with the process approach on the improvement of EFL learners’ writing performance specifically in relation to content/idea development and organization. The study employed a quasi-experimental research design with pre-and post-tests as well as the intervention which is the exposure to and use of automated writing evaluation (AWE) software with the process approach. Twenty-six Iraqi third year undergraduate students from the College of Education for Women at Baghdad University were assigned to receive automated feedback on their essays from the Criterion online writing evaluation system. Another 26 undergraduate students were assigned as a control group that received feedback from their writing instructor on their essays regarding content/idea development traditionally. According to the findings of the study, the incorporation of AWE software with process approach has a significant impact on students’ academic writing in term of improve content/idea development and organization. The experimental group outperformed the control group in their post-test results. This suggests that incorporating of process approach with AWE software was valuable in improving writing skills in term of content and organization. Each step of cognitive writing processes had overcome the barriers experienced by Iraqi EFL learners. They (Iraqi EFL learners) benefitted from Criterion features of planning ideas to develop essay content and


Introduction
Competent writers can improve the quality of their ideas and develop content as well as its organization through selecting suitable words that help readers understand the ideas they want to communicate.Therefore, to be efficient writers, learners need more practice, more writing tasks, and more constructive feedback for better learning and to provide more opportunities to revise and edit their writing based on that feedback (Shermis, 2014).Indeed, they must be aware of the writing and feedback processes needed to improve their writing performance (Hussin & Abdul Aziz, 2022;Wilson & Czik, 2016).This is usually the issue when it comes to EFL students in their writing performance.This is where insufficiently prepared student writers struggle when composing in academic contexts, often failing to adequately generate and develop ideas.
Composing text is considered to be one of the more difficult tasks in the ESL/EFL learning context, not just in school setting but also in higher education setting as well.In higher education scenarios where most of their academic reading and writing are conducted in English, students must perform well and gain higher-order knowledge to become capable of producing coherent text (Shermis, 2014).In addition, although EFL writing teachers evaluate their students' essays, they often neglect most stages of the writing process in favour of giving only corrective feedback on the form and function of the text and adopting the product approach where they focus on just giving the grades or marks for the written assignments or tasks instead of helping students develop their writing ability.EFL students need regular practice and feedback, not only corrective feedback on form or just grades.Therefore, teachers must rethink their approaches to writing instruction and adopt those that facilitate EFL students' mastery of writing skills as well as writing strategies to achieve satisfactory writing improvement (Wang & Wang, 2012).These problems are prevalent in Iraq, where there are big numbers of undergraduate students and writing instructors hardly respond to each student's written work.This is mostly because these teachers suffer from insufficient time and an increasing number of students in each class, and they are thus not capable of providing constructive feedback on students' writing, and the students struggle to produce writing due to few or no opportunities to practice, which leads to poor writing performance.

The Process Approach
The process approach derives from the cognitive process theory of writing.Hyland (2019) noted that "the process approach to teaching writing emphasizes the writer as an independent producer of text, but it goes further to address the issues of what teacher should do to help learners perform a writing task.There are numerous versions of this perspective but all of them recognize that writing depends on basic cognitive process that teachers should develop these by helping students to plan, define a rhetorical problem, propose solution and evaluate outcomes" (p.10).In a process approach, the essential parts of analysis are basically mental processes, like the stage of generating ideas and organizing information.Indeed, the process approach has been regarded as a way of thinking that assists students to analyse and organize their thoughts on related topics.What characterizes this approach is its focus on the idea that composing is achieved in stages that provide opportunities to manage and control the process of writing (Bayat, 2014).Since the 1970s, the process approach to teaching writing has linked feedback with revising and editing.Therefore, it is considered part of a writer's consequence of performance through the learning process that makes a difference in his/her writing performance (El Ebyary, 2010).It involves the stages of planning (generating ideas), translation (ideas are transformed into written text through grammar, syntax and spelling), and review (writers revise their work and correct any mistakes), which is under the control of the monitor (writers who are responsible of the changes based on input to achieve output) (Flower & Hayes, 2012;Mourssi, 2012).The process approach gives the students opportunities to plan, draft, revise, edit, and publish their writing (Souhila, 2015).Hasan and Akhand (2010)

explained further as follows
The process approach is based on the recognition of the writing process as cyclical, recursive or even disorderly rather than simple or linear.The focus shifts from the text to the writer, and the writer/students in our case, jump back and forth from one stage to another when they write.The second chapter of the mentioned book (Fifth Edition) titled as "The Writing Process" gives the students a complete idea of process approach.(p.82) In order to understand how the writer moves from one stage to another, there is a need to look at each one of these stages.In the planning stage, writers try to generate content for topics; therefore, content acts as a crucial construct of the writing process (Galbraith, 2009).In the field of EFL writing, learners experience challenges when they try to create content for topics (Faraj, 2015).Additionally, organizing ideas in ESL/EFL writing is considered a challenging task due to its key role in understanding the content of topics (Gabrielatos, 2002).ESL/EFL writers write for an aim; therefore, they must identify the discourse they need to produce to organize their thoughts efficiently and achieve that aim.Besides, previous studies have provided evidence on the challenges faced by ESL/EFL students regarding the organization of ideas and thoughts (Zakaria, 2016).A study conducted by Ruegg and Sugiyama (2013) claimed that the most difficult construct of writing is content/idea development and organization.Thus, the process of evaluating these constructs is challenging.Another study was conducted by Hsiao and Oxford (2002) stating that expressing ideas effectively requires organizational knowledge in order to achieve writing proficiency.

Criterion® Automated Writing Evaluation (AWE)
Criterion® is Automated Writing Evaluation (AWE) platform, designed by Educational Testing Services (ETS), for grading essays under determined conditions of testing (Casal, 2015, p. 148;Shrim, 2010, p. 22).The Criterion® Online Essay Evaluation Service provides automated essay scoring (summative assessment measures students' performance at the end of a semester) and formative feedback (conducted during the teaching process for the purpose of diagnosing students' difficulties and monitoring the writing process).In order to identify the various discourse elements, Criterion® as an automated writing evaluation (AWE) system can provide students with focused feedback on main aspects of writing like grammar and the mechanics and style, providing the same form of feedback writing teachers provide when grading students' essays (Burstein et al., 2014).The feedback is specific to the undergraduate students' essays and is based on the kinds of evaluations that writing teachers typically perform when assessing their writing.Criterion® is intended to be an aid, not a replacement, for classroom instruction.Its purpose is to ease the instructor's load, thereby enabling the instructor to give students more practice writing essays (Hyland & Hyland, 2019).It offers opportunities for students to submit their writing for immediate evaluation, receive individualized feedback, and interact through revising and editing based on feedback Criterion® has another feature, which is the students' ability to receive feedback from their writing instructors in addition to the automated feedback.This helps them focus not only on automatically detectable errors but also on other, subtler aspects of writing pointed out by the instructors.Second, it can provide feedback even in response to new essay prompts, which is not possible in other programs since their scoring capabilities are predetermined by training methods that use collections of essays scored by human raters.
Criterion® can provide feedback on the following constructs: syntax, discourse, topical content, lexical complexity, grammar, usage, mechanics, and style.This kind of feedback helps students to develop the discourse structures of undergraduate student writing.The software also offers a wide range of planning strategies to assist students in the process of planning their writing tasks.Criterion® can in fact be considered as a comprehensive automated portfolio administration system.Depending on the purpose of use and the stage in the writing process, teachers have the flexibility to impose a time limit or allow unlimited writing time (Calvo & Ellis, 2010).Burstein et al. (2014), Cotos (2014), and Shirmes ( 2013) have provided strong evidence that Criterion may influence instructors' teaching approaches.They revealed that when writing instructors adopted Criterion® for formative feedback, they found it was a suitable electronic technique despite differences in the way they integrated it.Furthermore, some writing instructors who adopted Criterion® observed that its detailed and accurate feedback on grammar and mechanics saved them class time and thus allowed them to focus on more complex issues like content development, organization, and coherence and cohesion.In their study, Chiu and Liao (2009) examined the use of AWE tools to detect errors and provide feedback on grammar errors reported by two AWE systems, Criterion and My Access.One hundred nineteen essays were graded, and feedback was provided on them by My Access, while 150 essays were graded by Criterion® on meaning, grammar, content and idea development, organization, language use and style.Results indicated that Criterion® performed better than My Access.Since My Access feedback had 10% accuracy in feedback targeting grammar, Criterion had a higher rate feedback accuracy (70-80%) in the same construct.
Writing instructors can manage writing tasks and writing assessment easily and provide focused feedback through using Criterion®" Moreover, it provides students with word processing, a spelling checker and mechanic's checker.In addition, Criterion® provides students with opportunities to receive immediate feedback on multiple drafts.Criterion can therefore be an aid in writing instruction but not an alternative to humans in writing instruction and writing assessment.Attali (2015) explained that Criterion® grades essays based on computing the features of written text.It depends mainly on analysing these features and provides explanations of its scoring system.Fundamentally, unlike human raters, AWE systems like Criterion are widely used in assessment as they can detect errors in grammar and vocabulary and grade essays as whole texts with diagnostic feedback on errors.
Indeed, currently research is increasingly being done in the area of automated writing evaluation systems in L1 and ESL contexts as compared to EFL contexts.For instance, a number of studies have been conducted to investigate the effectiveness of Criterion® on students' writing performance (Al Uthman, 2016;Chen & Cheng, 2008;El Ebyary, 2010;Liao, 2016;Shim, 2013), to examine the validity of corrective feedback on errors provided by Criterion® (Chaitanya, 2013;El Ebyary, 2010;Li & Lindsey, 2015), and to study the reliability and validity of Criterion® (Attali, 2015;Klobucar et al., 2013).From among the studies cited above, the study by Chen, Chu& Liao (2009) examined the use of the Criterion® and My Access AWE systems to detect and provide feedback on grammar.One hundred nineteen essays were graded by My Access and 150 essays were graded by Criterion on meaning, grammar, content and idea development, organization, language use and style.Feedback provided on all the essays by the two systems were submitted to performance analysis.The results indicated that Criterion® performed better than My Access.My Access showed 10% errors in feedback, while Criterion showed 70-80% in feedback accuracy.
However, although research has indicated the positive impacts of automated writing evaluation on writing improvement, little is known about the nature of such processes as providing formative feedback especially in EFL essay writing and more particularly in Arab countries, like Iraq.The authors believe that second and foreign language research writing pedagogy needs to provide sufficient opportunities for undergraduate students to practice writing through the adoption of automated writing evaluation (AWE), and the validity of this electronic technique needs to be improved beyond assessing grammatical and mechanical errors and expanded to the evaluation of how students express meaning, organize ideas and produce coherent written texts.

Research Purposes
This study examined the effects of Criterion® feedback with a process approach on Iraqi EFL undergraduate students' writing performance regarding content/idea development and organization.In this quasi-experimental study, the students received automated feedback from Criterion® that is embedded with the ideas of process approach to writing while control group participants received instructional feedback from their writing instructor.Thus, two research questions guided this research RQ1: How effective is the use of automated feedback software on Iraqi EFL undergraduate academic writing performance regarding content/idea development and organization?RQ2: Are there any significant differences between receiving automated feedback and receiving traditional feedback regarding content/idea development and organization?

Research Methodology
This study was conducted at the College of Education for Women, Baghdad University.This college prepares female students from various science and art disciplines to be English language teachers.One hundred eighty students volunteered to participate in the current study.From the 164 students who were eligible to participate, two groups of students were selected randomly (30 students for each group).Thirty students were assigned to receive automated feedback using the Criterion application with process approach to writing on content/idea development and organization, and 30 students were assigned to receive instructional feedback from their teacher.The researcher taught the experimental group with the assistance of Criterion® and process approach to writing and another writing teacher taught the control group without Criterion® support.Both writing instructors and students can access Criterion by clicking on the link http://criterion.ets.org/ and logging in with their usernames and passwords.Then they go to the assignment created by the instructor.Once they read the required topic, they go through the prewriting strategies in an idea tree.Then they can organize their thoughts and ideas, put everything together in an initial draft to be assessed by Criterion® and subsequently receive instant grades and automated feedback on writing constructs such as content/idea development and organization to enable them to correct any mistakes and develop their content.
Data were collected for this research from two groups through pre-and post-testing of both groups.Written data were then collected from the experimental group over a period of 10 weeks.The researcher asked these participants to submit their essays through the Criterion® website and make changes based on feedback received on each submission.The other part of the data was collected from the control group, who were asked to write essays and submit them to their writing instructor for evaluation.
In regards to the data analysis, particularly on testing the normality of data, Chou and Bentler (1995); Ghozali et al (2005) indicated that data are considered normally distributed if the Z-value skewness is less than ±3 and the Z-value kurtosis is less than ±7.In this research, the researcher followed the guideline suggested by Hair et al (2006) and thus set the critical cut-off value at ±2.58.All seventy-eight study samples (N = 52) were divided evenly into two groups.The control group consisted of 26 participants while the experimental group was also represented with the same number of participants (n = 26).As stated above, the study was conducted over a period of 10 weeks.Table 3 shows that the kurtosis and skewness results for every construct were within the range of ±2.58.The computed significance values were less than 0.05, indicative of an accurate test to determine if all the undergraduate students came from a normally distributed data.So, the skewness and kurtosis tests did not show evidence of non-normality and indicated two significant values.

Results and Findings
Effectiveness of automated feedback software (Criterion) with process approach on Iraqi EFL undergraduate academic writing performance (content/idea development) and organization The results revealed that giving automated feedback with process approach had a positive effect on the participating Iraqi EFL undergraduate students' academic writing performance in terms of the essay content of the experimental group (EG), as indicated by comparing the mean pre-test result (M = 14.00) with that of the post-test (M = 20.96).Thus, Iraqi EFL undergraduate students benefited from receiving automated feedback in developing their ideas regarding given topics through the stages of process of writing.The students' organization significantly improved as indicated by comparing the mean pre-test results (M = 13.1154) with that of the post-test (M = 16.2692).The Iraqi EFL undergraduate students also enhanced their vocabularies through using a wide range of organizational patterns and could organize their main and supporting ideas into paragraphs (e.g., introduction, body and conclusion) after receiving automated feedback on their writing.However, the results also indicated that adopting automated feedback with the process approach had a higher impact on content/idea development than on organization.

RQ2: Significant differences between receiving automated feedback and receiving traditional feedback (content/idea development and organization)
Table 2 shows that there was a statistically significant difference between the experimental group (EG) and control group (CG), as indicated by the results of a one-way ANOVA analysis (F(2,75) = 37.27,p = .000)on the content/idea development data.The EG demonstrated better performance than the CG with the highest mean result for idea/content development of 20.9615.In addition, the Tukey post hoc test showed that the statistical significance of the EG results was higher than that of the CG (p = .000)as shown in Table 3 This means that after receiving automated feedback information with process approach, the EFL undergraduate students improved their content/idea development, reaching their full potential by interacting positively with the automated feedback.Moreover, the undergraduate students' participation in the EG (i.e., receiving automated feedback with the process approach) had a positive effect on their writing ability as a result of the students devoting their efforts to improving their content and idea development based on criteria derived from the automated feedback.Overall, the undergraduate students became progressively more confident about developing their written content/ideas through their responses to automated feedback with process approach.
As shown in Table 2, there was a statistically significant difference between the EG and CG groups as demonstrated by the one-way ANOVA results (F(2,75) = 9.547, p = .000),which were, however, at a significance level lower than 0.05.In addition, the EG achieved better results than the control group (CG) as the mean for organization was greater at 16.269.Moreover, the Tukey post hoc test results (Table 3) showed that the EG results were of higher statistical significance than those of the CG (p = .000).However, the EG and CG results were both statically significant.Thus, it was found that the undergraduate students who received automated feedback on organization with the process approach outperformed those who received traditional instructor feedback.

Effects of Automated Feedback with the Process Approach
Providing feedback for EFL learners is a serious challenge for writing instructors as they must relay feedback from various sources or agents (e.g., teachers, peers, AWE systems or any other apps or systems).The study found that receiving feedback from Criterion and continually revising their written work with the ideas from the process approach has an impact on their writing performance in relation to content/idea development and organization.Prior to Iraqi EFL students dealing with criterion platform, they have to submit their essay as a final product without revision.When writing on criterion platform, students have more opportunities to overcome difficulties through revising their work.Thus, each cognitive process of writing had a crucial role in improving their writing behaviour through several stages of writing .It is worth mentioning that few studies have been conducted to examine the effectiveness of Criterion software on writers' organization and content/idea development.In a study conducted by Weigle (2010) examining the effects of scoring and feedback systems on the quality of writing claimed that Criterion is does not sufficiently provide assess the two main constructs of writing (i.e., content/idea development and organization).Xi (2010) claimed that recent studies confirmed that automated scoring and feedback support strategies useful in the writing process.This means that there is a link between adopting automated writing evaluation software and increasing writing skills.Based on cognitive process theory of writing, writing is not learned naturally but it requires more practice to improve writing skills and strategies.Thus, receiving Criterion feedback helped to reshape students' writing and improve their writing performance.It is apparently that revision strategy had been enhanced by receiving feedback generated from Criterion.In sum, Criterion has a positive impact if it is used to provide formative feedback in EFL writing classes in regular and appropriate ways.

Differences between Receiving Criterion® and Instructional Feedback
The current study revealed that giving Criterion feedback using the ideas from the process approach to writing has a positive effect on EFL writing performance.From instructional point of view, writing instruction had received a great potential in improving writing strategies (planning, revising and drafting) after combining process approach with automated feedback.Likewise, students benefited from receiving Criterion features; they listed their ideas ( planning), wrote first draft( drafting), revised based on Criterion feedback(revising).In this, they use writer's guide in Criterion platform to correct grammatical mistakes and returned to ideas list to add and support ideas to rewrite their essays.The findings related to the second research question in the present study are also consistent with previous research findings on the effect of automated feedback (Liu et al., 2016).For example, Liu et al (2016) examined whether adopting automated feedback via Coh-Metrix 3.0 would have an impact on Chinese college students' writing process strategies such as those used in revision.They found that generating indirect corrective feedback supported writing skills as well as strategies by spending more time to revise and correct mistakes.However, although the study was done using Criterion to provide automated feedback on writing constructs (e.g., content/idea development/organization), it differed from the current study in terms of focusing on seven constructs of writing.Still, taking into consideration writers' who depend on AWE systems, it has been proven that they produce writing with fewer grammatical and spelling mistakes with less attention to the content and organization (Hyland, 2019).The findings of current study are similar to Handayani, Rasyid & Lustyantie (2021) who investigated in what extant incorporating process approach with feedback has an impact on improving student' writing skills.They revealed that both process approach with feedback has significant effects on 35 students' writing performance.Recommendations

Recommendations for Practice
Study's findings provide practical recommendations for the integrating AWE applications in writing instruction.To prepare undergraduate students to engage in a writing process, they need to improve writing skills and strategies to become effective writers.Therefore, undergraduate students need an interdisciplinary knowledge to effectively engage in writing processes.The recommendation for practice was derived based on the current study's results.Feedback on EFL writing is undeniably significant in higher education, thus, it should be incorporated with technology due its capabilities to advance writing in all stages.Deploying AWE applications can improve not only writing skills and strategies but also improve writing instruction.Writing instructors should guide undergraduates plan their writing and choose use writing applications especially free ones as (Spell checker and Grammarly) to check their writing (essay or assignment).This research makes significant contributions to both our theoretical understanding of writing and its practical application in the context of EFL instruction.Theoretically, the study sheds light on the effectiveness of combining automated feedback with the process approach.It confirms that receiving feedback from Criterion and revising based on its prompts positively impacts both content/idea development and organization, two crucial aspects of effective writing.This finding complements existing research on automated feedback, further suggesting its potential for enhancing cognitive processes involved in writing.In the context of EFL instruction, the research offers practical recommendations for utilizing AWE applications like Criterion.Integrating automated feedback with process-oriented guidance helps undergraduate students improve their writing skills and strategies, particularly in planning, drafting, and revising.This resonates with previous studies indicating the benefits of AWE for revision and error correction.Additionally, the study highlights the importance of interdisciplinary knowledge and technology literacy for effective writing engagement, encouraging instructors to explore free AWE tools like spell checkers and Grammarly alongside traditional methods.Overall, this research offers valuable insights for both EFL scholars and instructors, suggesting that harnessing the strengths of AWE within a process-oriented framework can significantly benefit EFL writing instruction and enhance student writing performance.

Recommendations for Further Research
Based on the findings of the present study, future research should be conducted to investigate the effectiveness of automated feedback on other writing constructs.Moreover, research should be conducted on adopting other AWE systems as well as conducting studies to compare using these systems.The findings of these studies may draw attention to integrating technology in writing classes to ease writing instructors' workloads and to increase the levels of EFL undergraduate students' writing performance.An in-depth examination of how Criterion feedback impact undergraduate and postgraduate students' revision strategy.The impact of automated feedback on academic writing can be explored in further research, for example, comparing automated feedback generated by other AWE applications.Future work needs to add the instructor's readiness to adopt the latest innovation to increase the explanation of learners' writing performance.Also, it would be useful if we could gather qualitatively the participants' attitudes and experience for those who had positive and negative experiences with automated feedback.In a similar vein, further study could examine the impact of automated feedback on volunteers from different educational backgrounds.

Table 1
Mean and SDs for Writing Performance (Pre-& Post-Tests) (EG)