|
Megatester is a Web-based Computer Adaptive Testing system with the primary function of delivering multiple diagnostic versions of different kinds of tests, as well as providing robust test creation and administrative features. Developed for a company specializing in online training, located in California, USA.
When an examinee is taking a test on a computer, an estimate of the examinee's ability can be updated after each response. That ability estimate can then be used to decide the selection of the next questions. That is what Computer Adaptive Testing is.
CAT is much more efficient than a traditional paper-and-pencil test. Paper-and-pencil tests are typically "fixed-item" tests in which all the examinees answer the same questions. Since everyone is asked every question, all examinees are forced to answer many questions which are too easy or too difficult for them. These easy and hard items provide relatively little information about the examinee's ability level.
Megatester allows:
- tests creation, as well as using of multimedia content in them
- taking tests, controlled by CAT module
- detailed statistical reports
- administration of users and tests (including CAT parameters)
A test consists from a set of pages, where each page contains a question with multiple answers on it, one or more of which can be correct. Question can be a text, sound, Macromedia Shockwave or Flash demonstration. Each question and each answer have certain CAT scores (according to CAT, there are three types of scores: discrimination, difficulty and guessing), which later are used by CAT subsystem. The basic workflow is: when user logs into the system he gets a screen where he can choose either to take a new test or view the history of his previous tests. When a new test is chosen, user gets the next screen, where he can select a section of the test to take. Than, every following screen displays in browser a new question, which complexity depends on decision of the CAT-module, based on previous answers.
User's types
- User is able to take tests and see summary review
- Administrator is able to get permissions and create test creators, group administrators, and users and another administrators in the system, publish tests
- Test creator is able to create tests, but not to publish them
- Group administrator is able to create users in his group and make searching and other reviews on the whole his group
Functionality
Depending upon privilege, the user is able to see different screens. Functionality available according to privilege of logged user:
- User section
- CAT-based testing system
- Support of multiple types of questions
- Ability to continue an interrupted test
- Detailed results reporting, also in graphical form and with export of results in Excel and XML-files
- Animated tutorials to help the user use the system
- CAT- and 'questions type'-modules as independent add-ins to the system
- Administrator section
- Grouping of teachers, courses-creators and administrators
- Search for user score via name, test, date, group
- Ability to manage courses (create new, edit and delete existing)
- Tests creation section
- Creation of new tests, editing of existing tests and deleting of it.
- Upload on the server possibility for audio files and pictures.
- Teacher's section
- Examination of users, belonging to teacher's group, seeing their statistics and results, search capability.
- Checking users essay.
- Administration of users in teacher's group.
Basics of implementation
The system is using Microsoft Internet Information Server and Active Server Pages technology for dynamical HTML content generating. Browsers allowed: Netscape from version 3.0 and Internet Explorer from version 4.0. The content sent to user is a standard HTML version 1.3, with simple parts of JavaScript version 1.1 code, which are independent of browser platform.
The system on the server side is based on XML. We intensively used XML and XSL in this project, putting it from the beginning in the core of the whole architecture. Usage of XML allowed us to store tests (questions and answers) in a very flexible format, which can be read be different sub-systems of the system as well as by the CAT module. A sample of XML file for TOEFL test is here. For parsing of XML system uses Microsoft XML-Parser, version that is freely distributed with Microsoft Internet Explorer 5.0. That means the system requires Microsoft Internet Explorer 5.0 installed on the server.
Figure 1. The main schema of the system
The Web-browser sends a request to the web-server (Microsoft Internet Information Server). This request is processed by ASP script, implementing the core functionality of the system. The ASP script reads the XML file containing course questions and depending of which type of question is being processing, script uses the question template specified for this type module and generates HTML output to represent the question. The next question, which will be sent to the user, is dependant on the calculations done in the CAT-module.
If user exceeds given time limit to answer a question, an automatic submit from the browser to the server will be done and user will get the next question with no scores for the previous.
Mathematics
According to the Item Response Theory, probability for correct (u=1) answer for some user which true ability theta is defined could be calculated using formula (1).
Item information function:
To calculate new proficiency we are using iteration
where
until right side of equation (adjustment) becomes extremely small.
Test is finished when standard error becomes less than some value
Especially difficult task is items calibration.
Our implementation isn't flexible as necessary and doesn't work correctly
for all of three parameters of three-parameter IRT model. If you have any
investigation done in this area and would like to tail it with us, we will be very pleased.
Reports
The reporting section displays the results, plus allows user to go through his answered questions, see his answers and what the correct answer was.
User is able to see summary review in two forms: as table and as graphs.
- Tables
The results of the taken tests are presented as a table with question number, users score, maximal score value for this question, given time to answer the question and time taken by user.
By clicking on question number user gets the visual presentation of the question with the selected by user answer and the shown correct answer. If a question was sound based (Listening section), sound will be repeated.
- Graphs
There are possible two graphical representations of user's results.
1) Numbers of questions on the X-axis, scores as Y-axis, and two graphs: red graph - correct answers with maximum scores for it and blue graph - users answers and scores. By clicking on the question number on the X-axis user gets a screen with the selected question.
2) Numbers of questions on the X-axis, time as Y-axis (where the maximum is the maximal given time to answer some question), and one graph showing taken time to answer each question. In this way user can determine which questions where especially hard to answer for him. As above, by clicking on the question number on the X-axis user gets a screen with the selected question.
Used tools and technologies:
User Interface: HTML, DHTML, XML/XSL/XSLT, JavaScript
Server-side: ASP, ADO, C++/ATL (CAT component)
Web-servers: MS IIS 4.0
OS: Windows NT/2000
Time spent:
3 developers were working 12 months = 36 man-months.
|
