Home
###--> See project description for background and goals
I accomplished most of what I expected in the timeframe of the program. TypeOver has evolved dramatically over the past three months. Word prediction has came on leaps and bounds. We now have a bigram algorithm in place. This means that words relate to each other. For example, if the user enters the word 'to', word prediction would predict the word 'the' next. This speeds up typing so much.
For this post, I'll be showing you some of the developments we've made. First off, the user interface. Before GSoC, TypeOver had a rather mundane interface as shown here.
We did a lot of work and came up with a totally different design as shown here.
We did a lot more work on word prediction. Before GSoC, word prediction was quite dumb. Dumb as in it didn't relate words. For example, if you typed the word 'to', and then typed the letter 't', word prediction would predict the word 'to' again. This was because word prediction didn't have a clue you typed the word 'to' already. Now, after GSoC, word prediction takes the last word you typed into account.
iOS 7 has had a huge influence on the approach we took toward TypeOver. Apple introduced a lot more accessibility features including Switch Control. Switch Control is a system that lets switch users interact with and use the entire iOS operating system. Up until this, various companies including Komodo OpenLab had found a way to use VoiceOver (Apple's screen reader for blind people) to enable switch users to use iOS. Switch Control is quite different from VoiceOver which meant we had to adapt TypeOver to work with it as well as VoiceOver. If the user is using Switch Control, when they select a key, a subview pops up with individual keys for all the letters on that selected key.
All in all, I really enjoyed working on this wonderful project and I'm a better programmer for it. I would like to thank Jorge Silva, Mauricio Meza and Tom Nantais for giving me this opportunity.
This week I focused mainly on word prediction, specifically user-added words. As it stands, TypeOver let's users add new words to a database that is separate from the stock word prediction database. These words then appear before the stock words in prediction. This can get really annoying because there's no way users are going to use the added words more than function words like 'the'. So, we need to create a balance. To do this, user-added words will have to have a frequency. The difference is, the stock words' frequencies don't change, whereas the user-added words' do. So, when a word is added, its frequency is one. Then, when the user uses that word, the frequency increases to two and so on. The problem with this is, stock words have frequencies that are in the hundreds. So, when the prediction algorithm is happening we will add a starting frequency to each user-added word on top of the original frequency. This should give the user-added words a chance to compete with stock words for prediction places. The starting frequency will be the frequency of the two hundredth stock word. This should give us something to test anyway. I have everything implemented and modified except for the prediction algorithm.
Also this week, I added a squash effect to the keypad keys. This makes the radial gradient more elliptic.
This week I was off from Wednesday until Sunday. That being said, on Monday, the first alpha release went out to testers. This is a big step forward. To my knowledge, there are four testers. Their input will help us tailor TypeOver to the needs of actual switch users.
Also on Monday and Tuesday, I continued to implement the user interface. I added a very subtle black border to the keypad keys. The border just makes the keypad clearer and definitive. I'm following the PDF from the designer to the letter.
I also fixed a few bugs that were introduced when I implemented the new layout.
This week I continued to implement the user interface for the application. The designer has provided a PDF that details exactly how the interface will look and behave. So, I have been following that to implement the interface. On Monday, I created a radial gradient in the custom button class. The gradient adds a really nice effect to each button using this class. The gradient is added using core graphics. Luckily, core graphics has a built-in method to create a radial gradient so this was quite simple to implement once I sorted it out in my head.
Later in the week, I implemented an entirely new layout for the keypad in the main view of the application. This layout was specified in the PDF. I didn't use Interface Builder, I did it all in code. I chose this method because I find it much easier to get controls to the exact size and origin that's needed using code. Unfortunately, the designer didn't factor in some of the buttons so I had to get a bit creative. I think it's pretty good but it can easily be modified.
Unfortunately, this week Alpha 1 didn't go out. We hope it'll ship early next week.
This week was a big one. We are making preparations for the first alpha release that will likely take place early next week. The alpha will have some restrictions. For example, you won't be able to add new words to prediction. We just don't feel like this feature is ready for testers, so I commented out the code that displayed the 'add word to dictionary' button. Also, obviously this build doesn't have any speech capabilities as this hasn't been implemented yet. Also, I made some minor user interface tweaks for the alpha.
Also this week, just because we aren't including the ability to add new words to prediction in the alpha build didn't mean that I couldn't continue writing the code for it. To make it as clear as possible for the user, the 'add word to dictionary' button now states the word it'll add. For example, it would say 'add "Tecla" to dictionary' on the button if 'Tecla' was the word in question.
I also wrote a small application to test a new text-to-speech engine this week. I cannot state what engine this was because it's in beta and I signed a non-disclosure. I can say that it's a strong contender for TypeOver though.
This week I started to implement the user interface for TypeOver. Until now, it had a rather mundane interface that I had implemented at the start. So, I started with the buttons. They were grey before and turned yellow when the VoiceOver cursor was on them. These backgrounds were just PNG images. I completely deleted the images and wrote a piece of code to create new images using CoreGraphics. These images are then used as button backgrounds. Currently, the colour scheme is black and blue; blue when the VoiceOver cursor is on the button, black otherwise.
Also this week, I implemented the ability to add new words to prediction. The new words go into a separate database. When the user types a word that's not in prediction, a button appears under the text view. The button let's you add the word to prediction. The SQL query for prediction now gets the user added words before the words from normal prediction. This may need to be changed at a later date because it could get annoying for users when their added words appear first. But this will likely be determined by beta testers.
On Thursday, the team got together with the designer. We discussed the user interface mostly; placement of buttons, etc.
I also fixed some minor issues this week.
This week I focused mainly on fixing bugs that were introduced when the bigram model was implemented. That threw off a lot of the code that dealt with various issues. One example would be the fact that SQL doesn't recognize apostrophes as they are. It needs to see two apostrophes like 'don''t' as opposed to 'don't'. I presume it's a syntax issue. Anyway, I had written code to deal with such an event but it was rendered null and void by the method I wrote last week that parses the text you wrote and gets all the variables (explained in last week's post). So, I thought about it and came up with a very simple, clean solution. Objective C has a built-in method to replace an occurrence of a string within a string with another string. For example, you could replace one apostrophe with two. That is what I did to solve the problem.
This week I want to talk a bit about the future of the project in terms of what's next. Word prediction seems to be working well. The current bigram corpus may need to be replaced with a more up-to-date corpus but that still remains to be seen.
The current UI is going to be re-designed. We are working with a designer and they have came up with a beautiful black and blue design.
A big feature that is still not implemented is text-to-speech. We tried OpenEars but had issues with clarity. (OpenEars is an open source text-to-speech engine for iOS.) We are looking at alternatives. We are also exploring some new features in the upcoming iOS 7 to ensure TypeOver is compatible.
This week I continued to improve the implementation of the bigram model. Until now, all the variables for word prediction were being updated when the user entered a letter. This caused some issues, including the fact that if you entered a space and then backspaced, word prediction wouldn't register that you were back at the previous word thus generating wrong predictions. So, I wrote a method to parse the text you wrote to get three strings; the current word, the previous word and the word delimiter (string in between current word and previous word). If the word delimiter is anything other than " " or ", ", then bigram data is irrelevant. The method cycles backwards through the string and checks if each character is a delimiter. It passes each character to a function that returns a boolean. This boolean will be true if the character is a delimiter. This solves the problem because the method is getting live, up-to-date data from the text area.
Also this week, I fixed some minor issues.
This week I made some major progress on the bigram model in word prediction. At the start of the week, I adapted the code that generates the word prediction database to read from a text file and insert bigram data into a new table. The code runs through all the lines in the text file one by one. The lines are in the format 'first word, next word, bigram frequency'. The code I wrote gets the ID for each of the two words from the main dictionary table, then inserts a row into the bigram table using that data, i.e., ID1, ID2, bigram frequency. I also created an index on the new table. The index uses the ID1 and frequency columns. This will make the query faster in TypeOver.
Then, I adapted the code in TypeOver to take advantage of the bigram data. The user selects a word from prediction, then the code runs a query to get the ID for that previous word, 'wordId'. Then, the next set of word prediction results are based on that word. The SQL behind this is pretty simple. If wordId is empty, the SQL query is the same as before. But, if the wordId has a value, the query returns the rows with a matching 'ID1' column in the bigram table. It returns the 'ID2' column and matches the IDs with words in the main dictionary table. Also, it orders the results by descending bigram frequency.
There will be times when there won't be enough bigrams for a word to fill prediction. In such circumstances, we want to combine bigram with unigram results. I wrote an algorithm to deal with such an event. Basically, the algorithm gets the bigram results first. Then, it subtracts the number of results from 8 (number of prediction buttons) and gets unigram results for the remaining number, while making sure not to duplicate words that were in the bigram results.
Later in the week, Tom, my mentor, found a bigram corpus online. Up until this, I was using a text file with just a couple of bigrams that I wrote. With the new corpus, I can more effectively test my algorithm. We are unsure whether this is the corpus we will permanently use as it is a little outdated.
With the new bigram model in place, word prediction is much faster. Say you want to type 'I want a drink please'. Before, you may have had to type 'wa' to get 'want'. Now, it's likely you wouldn't have to type anything. The algorithm would take the 'I' into account, and predict words relative to it. On an old build without the bigram model the prediction list for "I w" was: "was, with, were, which, we, who, when, would" and for the new build it was: "was, would, want, will, went, wish, wanted, were".
Also this week, I added the option to increase or decrease the font size.
This week I implemented some new stuff in TypeOver. You can now just type key letters from the word you want and the word prediction algorithm can guess, remarkably accurately, what you want. I achieved this by using the wildcard string option in SQL. The word prediction algorithm inserts a wildcard string ('%') after every letter. This is done using a basic while statement. 'Text speak prediction' can be toggled in Settings.
I also implemented navigation controllers in the application storyboards. I have been meaning to do this from day one. In my view, navigation controllers provide a much cleaner, cohesive experience.
Also, this week I fixed a few bugs.
This week I continued learning about SQL, specifically the Join clause. The Join clause is very useful for querying data in two different tables at once. Let's say you have a table called 'Customers' and in that table, a column called 'AreaCode'. The 'AreaCode' column can just be a reference for an ID in another table. Let's call that other table 'CustomerInfo'. This table has a column called, again, 'AreaCode'. This column contains the actual area code. So, now we can run a query using the Join clause that returns the 'Customers' table and the 'CustomerInfo' table side by side. This method is very useful because if you need to change the area code you only have to amend one record rather than potentially thousands.
Also this week, I started learning plain C. Well, more practising than learning. I needed to brush up on C conditionals, loops, functions etc.
These links helped me this week:
- http://thenewboston.org/list.php?cat=14 - this helped me with the plain C
- http://www.youtube.com/watch?v=lZ3FTPQssu4 & http://www.youtube.com/watch?v=4_YllmGseko - this two part video helped me with the SQL Join clause
This week I continued learning about the technologies I'm going to be using in the project, specifically SQL. I furthered my knowledge about the current SQL implementation. I now know exactly how the SQL query works in TypeOver. The user enters a letter and this calls a function. This is an array function which passes in a string as a parameter. This string is made up of the first letters of the word desired. The user has entered these letters. SQL queries this string in the word prediction database. Then, the results are populated into an array. The function then returns this array.
- [strQuery appendString:@"SELECT * FROM WORDS WHERE WORD LIKE '"];
- [strQuery appendString:strContext];
- [strQuery appendString:@"%' ORDER BY FREQUENCY DESC LIMIT 10;"];
The above code is the current SQL query in TypeOver. The first line selects everything from the table 'WORDS' and searches it for words like the string passed in on the second line. Then, the third line just sorts the results by descending frequency and only allows ten.
Also, I explored how the actual database was created using the XMLtoSQLite project in the repository. At first, the database was in XML format. The project uses Objective C's built-in XMLParser class to parse the data. The data is then run through and inserted into the database. But before any of this, the database is opened and an index is created. The index sorts the data by descending frequency. This just speeds up queries.
This week I also wrote a small OS X application that loads a database and allows users to input data. This was really to just get comfortable with SQL and databases. It also helped me understand the current SQL in the project.
Also, this week I implemented the ability for users to send SMS and iMessages from within the application. This means that the user can quickly create messages and send them without leaving TypeOver.
This week I implemented the ability for users to set the dwell time using their Tecla Shield or other switch interface. This will provide a sense of unity between the switch interface and the application. Also, if the user prefers to set the dwell time manually, this is possible in Settings.
Also, this week I started learning SQL. Currently, the project uses SQLite for word prediction. The database is based on a unigram model. I am going to implement a bigram model. This will provide a relationship between words. For example, 'the' would be followed by 'theatre' in word prediction. But, this week I have been learning the concepts of SQL and databases.
The current word prediction model is unigram. There is no relationship between words. The database was brought over from the Tecla Access application for Android. The file was in XML format so we created an OS X application that parsed the XML and organised the data into an SQLite database. This database was then installed into TypeOver. When TypeOver launches, the database opens. Every time the user enters a letter, an SQL query runs. The SQL query returns words beginning with the letters entered and orders them by descending frequency. This leaves the most frequent words at the top of word prediction. The bigram model would vastly speed up typing.
These are the resources I used to understand SQL:
- http://www.sqlite.org/queryplanner.html - this helped me understand the current SQL query in TypeOver
- http://www.youtube.com/watch?v=z8136vWng3c - this video helped me understand the core fundamentals of databases
- http://www.youtube.com/watch?v=z8136vWng3c&list=PLLk083vNjCXYTT_oLTDlGq2gelsILnLIT - this video series helped me understand the Objective-C/C SQLite API