NAND writing is now semi-reliable (although one has to be VERY careful not to interrupt the device in the middle of a write operation), but it is enough to have something akin to a full-functional OS, backed by non-volatile storage.
People interested in the project should be familiar with the myriads of Linux "distributions" floating around. An operating system consists of two major domains: one is the kernel, which is what manages the hardware, and one is the userland which contains things like shells and other UIs, package managers, etc. Software that help users install and run useful programs. Ubuntu is a popular distribution that I run on my personal machine. Android could also be considered a distribution (though I believe it has some apparently messy kernel patches).
I decided that Debian would be an interesting thing to try, since we would then instantly have a userland and a pool of ready-compiled applications. Using a slightly dated root filesystem here: http://lists.debian.org/debian-arm/2007/01/msg00034.html, a initrd and further kernel configurations were sufficient to get it to run. Thus, we can now compile programs for iPhone Linux on iPhone Linux. The process is rather slow due to the processor and inefficient NAND device driver (pending a real FTL), but at least theoretically, iPhone Linux is now self-hosting.
This should be pretty much enough for those who are more into the userland development side of things to come in, possibly using Debian as a base to build anything else (as I believe it is standard enough).
I will be offering instructions on how to get this all to work soon. The (modified for gadget serial terminal) rootfs is fairly hefty (around 130 MB), so I'm not sure how we'll handle distribution of that.
Sunday, December 14, 2008
Saturday, December 6, 2008
Porting drivers to Linux
We've made some progress on the USB gadget driver for Linux, and we're now running a generic serial gadget for communication. This implementation is important because USB is now a lot less laggy and things like ethernet over USB, etc., can eventually be supported, easing access.
We've also got pretty far with porting the NAND driver to Linux. Most of the read support is now there, and we've isolated the routines in the iPhone kernel where the raw hardware write occurs. CPICH and c1de0x are working on reversing it. Hopefully, it will be analogous enough to reads that it won't take a huge amount of time to work out.
This is different from reversing their FTL, however, which is a complicated slew of data structures, merge buffers and other exotic algorithms that take care of evenly distributing writes throughout the device and also making writes take less time.
I think reversing all of that would take too much time and effort. Instead, my proposal is to just reverse the hardware NAND writes. Instead of using a partition, we would have a loop-mounted root filesystem (similar to how Wubi is setup), with the root filesystem being a file on the Media partition. Since there's a non-empty file at that location, the FTL system, whatever it is, must create a one-to-one mapping from logical sectors to physical NAND pages. We can already read the mapping it creates (we have already reversed the read-side FTL code), and so all we have to do to alter the data is to write to the same pages we would've read from. Of course, this means that wear-leveling and bad block handling is not performed. However, if we use a filesystem that's aware of bad blocks and can wear-level (YAFFS or JFFS2), then it amounts to the same thing. The wear-leveling would then take place over the particular physical pages belonging to the rootfs image, rather than the entirety of the NAND. This would make the physical pages belonging to the rootfs image wear out a little faster than the rest of the NAND, but the actual effect of this should be inconsequential.
The additional benefit of this setup is that there's no repartitioning required, so setup is cinch. See this wiki document for specific proposed implementation details.
We've also got pretty far with porting the NAND driver to Linux. Most of the read support is now there, and we've isolated the routines in the iPhone kernel where the raw hardware write occurs. CPICH and c1de0x are working on reversing it. Hopefully, it will be analogous enough to reads that it won't take a huge amount of time to work out.
This is different from reversing their FTL, however, which is a complicated slew of data structures, merge buffers and other exotic algorithms that take care of evenly distributing writes throughout the device and also making writes take less time.
I think reversing all of that would take too much time and effort. Instead, my proposal is to just reverse the hardware NAND writes. Instead of using a partition, we would have a loop-mounted root filesystem (similar to how Wubi is setup), with the root filesystem being a file on the Media partition. Since there's a non-empty file at that location, the FTL system, whatever it is, must create a one-to-one mapping from logical sectors to physical NAND pages. We can already read the mapping it creates (we have already reversed the read-side FTL code), and so all we have to do to alter the data is to write to the same pages we would've read from. Of course, this means that wear-leveling and bad block handling is not performed. However, if we use a filesystem that's aware of bad blocks and can wear-level (YAFFS or JFFS2), then it amounts to the same thing. The wear-leveling would then take place over the particular physical pages belonging to the rootfs image, rather than the entirety of the NAND. This would make the physical pages belonging to the rootfs image wear out a little faster than the rest of the NAND, but the actual effect of this should be inconsequential.
The additional benefit of this setup is that there's no repartitioning required, so setup is cinch. See this wiki document for specific proposed implementation details.
Sunday, November 30, 2008
Donations...
A couple of people asked for a donate button of some sort. I hesitate because of the sudden complexity of things when money is involved. Here is what I think would be best:
- Donate your time and skills if you can rather than money.
- If you cannot do so, please consider sponsoring something we could directly use: Such as a USB serial cable (about $45 for parts from Sparkfun), or perhaps upgrading the Slicehost slice we're borrowing from pumpkin ($60 per month, though it would only be useful if we could get enough money for at least six months), used or broken devices for testing, etc. Or something like a pizza for one of our contributors on a late night hacking binge (we will distribute food donations on a round-robin basis =P)
- My paypal address is my gmail e-mail address (planetbeing). If you do make a donation, please specify exactly what it is for. A gigantic slush fund is something that I wish to avoid.
Saturday, November 29, 2008
Updating and uninstalling openiboot
It's actually slightly problematic to use the "Update Firmware" feature of iTunes with openiboot installed, so I updated openiboot with an uninstall facility.
I uploaded the binaries and instructions necessary to update openiboot (and then uninstall it if you wish) to: http://www.iphone-dev.org/planetbeing/openiboot-uninstall.tar.gz
I uploaded the binaries and instructions necessary to update openiboot (and then uninstall it if you wish) to: http://www.iphone-dev.org/planetbeing/openiboot-uninstall.tar.gz
Friday, November 28, 2008
Linux on the iPhone!
I'm pleased to announce that the Linux 2.6 kernel has been ported to Apple's iPhone platform, with support for the first and second generation iPhones as well as the first generation iPod touch. This is a rough first draft of the port, and many drivers are still missing, but it's enough that a real alternative operating system is running on the iPhone.
What we have:
- Framebuffer driver
- Serial driver
- Serial over USB driver
- Interrupts, MMU, clock, etc.
What we have in openiboot (but hasn't been ported yet):
- Read-only support for the NAND
What we don't have (yet!):
- Write support for the NAND
- Wireless networking
- Touchscreen
- Sound
- Accelerometer
- Baseband support
The current userland we're using, in the interest of expedience, is a Busybox installation created with buildroot, but glibc works fine as well, and we're going to build a more permanent userland solution.
A demonstration video can be seen here: http://www.vimeo.com/2373142
Instructions here: http://www.iphone-dev.org/planetbeing/LINUX-README.txt
Download here: http://91.186.26.18/iphone/files/iphonelinux-demo.tar.gz (look for mirrors in the comments)
EDIT: The instructions are missing the step that you have to select openiboot console from the menu before performing the "sudo ./oibc" step. Just be aware you have to do that if it seems like you're not getting a response from the oibc client.
Project lead: planetbeing
Contributors: CPICH, cmw, poorlad, ius, saurik
If you're experienced with hacking/porting Linux and especially if you're experienced with porting Android, I'd definitely like to hear from you. Come chill in the #iphonelinux channel on irc.osx86.hu. Thanks. :)
EDIT: I was asked a couple times by people who wanted to donate (financially) to the project. I made a post discussing this possibility, if you are interested.
What we have:
- Framebuffer driver
- Serial driver
- Serial over USB driver
- Interrupts, MMU, clock, etc.
What we have in openiboot (but hasn't been ported yet):
- Read-only support for the NAND
What we don't have (yet!):
- Write support for the NAND
- Wireless networking
- Touchscreen
- Sound
- Accelerometer
- Baseband support
The current userland we're using, in the interest of expedience, is a Busybox installation created with buildroot, but glibc works fine as well, and we're going to build a more permanent userland solution.
A demonstration video can be seen here: http://www.vimeo.com/2373142
Instructions here: http://www.iphone-dev.org/planetbeing/LINUX-README.txt
Download here: http://91.186.26.18/iphone/files/iphonelinux-demo.tar.gz (look for mirrors in the comments)
EDIT: The instructions are missing the step that you have to select openiboot console from the menu before performing the "sudo ./oibc" step. Just be aware you have to do that if it seems like you're not getting a response from the oibc client.
Project lead: planetbeing
Contributors: CPICH, cmw, poorlad, ius, saurik
If you're experienced with hacking/porting Linux and especially if you're experienced with porting Android, I'd definitely like to hear from you. Come chill in the #iphonelinux channel on irc.osx86.hu. Thanks. :)
EDIT: I was asked a couple times by people who wanted to donate (financially) to the project. I made a post discussing this possibility, if you are interested.
Tuesday, November 18, 2008
Poorlad's menu implemented; Porting issues resolved
Yesterday night, I merged in a branch I was working on for poorlad's menu. A version of that beautiful menu is now in Git. His menu included a version string at the bottom. We didn't have any way to keep track of versions and builds before, so this was actually a good idea that I had to implement. Because I didn't want to implement support for non-fixed width fonts, or add another space-consuming font, I just used the console font I was already using for that part. I also had to brighten the gradient on the bottom of the screen, since it was basically invisible due to gamma issues otherwise.
The border between the gradient and the "black" is clearly visible on my device. This is probably because of a gamma issue. When poorlad comes back, we can ask him to calibrate it more.
Otherwise, it looks pretty good! In order to make this possible, I added in stb_images.c, a great tiny little image library that can read PNG, JPEGs and even PSD files and does zlib decompression as an added bonus. This will be a great help if we decide to change things or need to add more stuff that consumes a lot of space. I also added in a basic function to perform alpha blending (albeit comparatively slowly).
Sadly, while I was busy making these changes, ius from IRC actually begun to implement poorlad's menu without me knowing about it, so we ened up duplicating each other's efforts. He was able to compile in zlib and libpng, but the cost was to inflate the final binary to 347 KB. Whereas taking out the old menu images, and adding small, compressed PNGs and the stb_images library instead actually made openiboot smaller than it was before! His decision to preblend the images, rather than attempt alpha blending on the device, was probably more optimal from a performance perspective.
Steven Troughton-Smith told me on Twitter that he has actually implemented his own boot menu as well. I'm not sure if he used the new PNG code or not, but the new code makes it pretty easy for a competent programmer to add in whatever menu they would like. I'd tell everyone to skin away, but we should keep as few wild branches of this project as possible, since everyone randomly installing openiboot just for kicks (especially a modified version) and then coming to us (read: me, ultimately) for support is something we don't have the resources to handle at this moment.
On the porting side, the issues with installation, optimizing NOR access on iPhone 3G, NAND access on a few devices all seem to have been fixed, so we can basically scratch the first two items off of the list I put up in the last post. I'm pleasantly surprised at how relatively easy it was.
Anyway, now for the kernel. Well, if I don't get distracted by writing to NAND.
The border between the gradient and the "black" is clearly visible on my device. This is probably because of a gamma issue. When poorlad comes back, we can ask him to calibrate it more.
Otherwise, it looks pretty good! In order to make this possible, I added in stb_images.c, a great tiny little image library that can read PNG, JPEGs and even PSD files and does zlib decompression as an added bonus. This will be a great help if we decide to change things or need to add more stuff that consumes a lot of space. I also added in a basic function to perform alpha blending (albeit comparatively slowly).
Sadly, while I was busy making these changes, ius from IRC actually begun to implement poorlad's menu without me knowing about it, so we ened up duplicating each other's efforts. He was able to compile in zlib and libpng, but the cost was to inflate the final binary to 347 KB. Whereas taking out the old menu images, and adding small, compressed PNGs and the stb_images library instead actually made openiboot smaller than it was before! His decision to preblend the images, rather than attempt alpha blending on the device, was probably more optimal from a performance perspective.
Steven Troughton-Smith told me on Twitter that he has actually implemented his own boot menu as well. I'm not sure if he used the new PNG code or not, but the new code makes it pretty easy for a competent programmer to add in whatever menu they would like. I'd tell everyone to skin away, but we should keep as few wild branches of this project as possible, since everyone randomly installing openiboot just for kicks (especially a modified version) and then coming to us (read: me, ultimately) for support is something we don't have the resources to handle at this moment.
On the porting side, the issues with installation, optimizing NOR access on iPhone 3G, NAND access on a few devices all seem to have been fixed, so we can basically scratch the first two items off of the list I put up in the last post. I'm pleasantly surprised at how relatively easy it was.
Anyway, now for the kernel. Well, if I don't get distracted by writing to NAND.
Sunday, November 16, 2008
Porting to iPhone 3G and iPod touch
Hey guys,
The lack of updates for the past few days is because many of you decided to visit us in IRC, thus enabling work to be done on porting openiboot to the iPod touch and the iPhone 3G (in particular because I don't have an iPod touch at the moment).
I'm pleased to report that everything now seems to be working on the iPhone 2G and the iPhone 3G (albeit NOR read/write on the iPhone 3G is unoptimized and is unacceptably slow). There is apparently an outstanding issue with the NAND ECC on some (?) iPod touchs, and also some people can't seem to actually install openiboot to NOR on both iPhone 2G and iPod touch. Unfortunately, the problem is that these things happen on devices that I don't have physical access to, and IRC is often a frustrating medium for communicating with testers. I'm confident these issues will be resolved soon, though.
So, current simultaneous projects:
1. Resolve openiboot porting issues
2. Implement poorlad's boot menu
3. Work on write support for FTL
After at least one of those things are done, we'll be working on the Linux kernel.
The lack of updates for the past few days is because many of you decided to visit us in IRC, thus enabling work to be done on porting openiboot to the iPod touch and the iPhone 3G (in particular because I don't have an iPod touch at the moment).
I'm pleased to report that everything now seems to be working on the iPhone 2G and the iPhone 3G (albeit NOR read/write on the iPhone 3G is unoptimized and is unacceptably slow). There is apparently an outstanding issue with the NAND ECC on some (?) iPod touchs, and also some people can't seem to actually install openiboot to NOR on both iPhone 2G and iPod touch. Unfortunately, the problem is that these things happen on devices that I don't have physical access to, and IRC is often a frustrating medium for communicating with testers. I'm confident these issues will be resolved soon, though.
So, current simultaneous projects:
1. Resolve openiboot porting issues
2. Implement poorlad's boot menu
3. Work on write support for FTL
After at least one of those things are done, we'll be working on the Linux kernel.
Wednesday, November 12, 2008
NAND filesystem now readable!
Amazingly enough, the FTL_Read stuff from last night was pretty much correct! After that, it was relatively trivial to port over the HFS+ code I've already written (which was in pure C... finally that [fail] design decision has been vindicated =P).
As you can see in the screenshot below, with the latest Git revision, you can browse the filesystem from openiboot!
Next on the list is to port openiboot over to the iPod touch and iPhone 3G. It's probably just a matter of putting in different numbers for the GPIO ports, but we'll see.
After that, I will implement poorlad's bootmenu (which everyone seems to like).
After that, well... We have pretty much all the devices now, so we'll start looking at the Linux kernel. If you're a Linux kernel guy who would be willing to help (preferrably you have experience porting Linux to new ARM platforms), please leave a comment here. I can do most of the muscle work, but it'd be nice if someone can show me how to set up the source tree properly for the new port.
As you can see in the screenshot below, with the latest Git revision, you can browse the filesystem from openiboot!
Next on the list is to port openiboot over to the iPod touch and iPhone 3G. It's probably just a matter of putting in different numbers for the GPIO ports, but we'll see.
After that, I will implement poorlad's bootmenu (which everyone seems to like).
After that, well... We have pretty much all the devices now, so we'll start looking at the Linux kernel. If you're a Linux kernel guy who would be willing to help (preferrably you have experience porting Linux to new ARM platforms), please leave a comment here. I can do most of the muscle work, but it'd be nice if someone can show me how to set up the source tree properly for the new port.
Tuesday, November 11, 2008
FTL
I don't know how I was talked into reversing a FTL, but we're actually on our way. I've managed to enlist the aid of CPICH (who has been helping with the lower layers as well, he's our human HexRays) and just recently, pumpkin, who you will know from the Dev Team. pumpkin will be the heavy support that's necessary to take down _FTLRestore, which is the most complex function I've seen in 1.1.4 iBoot (and I've pretty much have seen all of it). pumpkin is very good, so this task should be now be cut down to "fairly difficult" from "completely impossible".
The strategy so far has been me methodically hacking through the functions in the order that they are called, completely decompiling them, understanding them, and assimilating them into openiboot. Toward this end, I've been working on FTL_Open, which is a fairly large (but as it turns out, boring) function, but has been useful in enlightening us on several of the large data structures FTL uses.
Meanwhile, CPICH works on functions ahead of me, so that when I reach them, a lot of the thorny underbrush has been cleared out and my job becomes much easier and faster. Toward this end, he has been working on FTL_Read, which uses the data structures that the now-completed FTL_Open should populate.
_FTLRestore is sort of a "bonus", since it's not normally called if the iPhone was shut down normally and everything is cleaned up. However, since recovering faulty data structures require all redundancies to be exploited, reversing this would let us gain a lot of insight into how the FTL works. It's also, naturally, an enormously complex function, and hence I wisely delegated it to pumpkin. =P (We will probably end up working on it together)
The one thing that troubled me was that the code we were reversing is for 1.1.4 whereas we primarily need it to work on 2.0. However, due to the fact that I had it better mapped out than the 2.0 iBoot, and the fact that the equivalent 2.0 code was much more complex (lots of function pointers flying around, and a weird switch idiom I haven't quite figured out yet), We decided to stick to the 1.1.4 iBoot.
After completing FTL_Open, I had a bit of a panic when I discovered it did not work at all on my 2.1 phone, and I could not find any obvious bugs with it. This might've meant that all our work on 1.1.4's FTL was for naught. Forgoing sleep, I tore through the 2.1 iBoot, locating the analogues to my already reversed 1.1.4 functions (I had given up trying to trace through the function pointers the first time around), and called them directly with my special version of iBoot (patched so that one of the commands was able to call arbitrary iBoot functions with arbitrary arguments). I managed to find a couple of bugs with my VFL code, and after having fixed them, FTL_Open appears to have worked. I think. It just finds and reads several data structures from NAND. It remains to be seen if I'm even reading the right thing.
Now for some sleep.
The strategy so far has been me methodically hacking through the functions in the order that they are called, completely decompiling them, understanding them, and assimilating them into openiboot. Toward this end, I've been working on FTL_Open, which is a fairly large (but as it turns out, boring) function, but has been useful in enlightening us on several of the large data structures FTL uses.
Meanwhile, CPICH works on functions ahead of me, so that when I reach them, a lot of the thorny underbrush has been cleared out and my job becomes much easier and faster. Toward this end, he has been working on FTL_Read, which uses the data structures that the now-completed FTL_Open should populate.
_FTLRestore is sort of a "bonus", since it's not normally called if the iPhone was shut down normally and everything is cleaned up. However, since recovering faulty data structures require all redundancies to be exploited, reversing this would let us gain a lot of insight into how the FTL works. It's also, naturally, an enormously complex function, and hence I wisely delegated it to pumpkin. =P (We will probably end up working on it together)
The one thing that troubled me was that the code we were reversing is for 1.1.4 whereas we primarily need it to work on 2.0. However, due to the fact that I had it better mapped out than the 2.0 iBoot, and the fact that the equivalent 2.0 code was much more complex (lots of function pointers flying around, and a weird switch idiom I haven't quite figured out yet), We decided to stick to the 1.1.4 iBoot.
After completing FTL_Open, I had a bit of a panic when I discovered it did not work at all on my 2.1 phone, and I could not find any obvious bugs with it. This might've meant that all our work on 1.1.4's FTL was for naught. Forgoing sleep, I tore through the 2.1 iBoot, locating the analogues to my already reversed 1.1.4 functions (I had given up trying to trace through the function pointers the first time around), and called them directly with my special version of iBoot (patched so that one of the commands was able to call arbitrary iBoot functions with arbitrary arguments). I managed to find a couple of bugs with my VFL code, and after having fixed them, FTL_Open appears to have worked. I think. It just finds and reads several data structures from NAND. It remains to be seen if I'm even reading the right thing.
Now for some sleep.
Saturday, November 8, 2008
Why iPhone Linux?
This is a post I wrote a long time ago, when this blog was first conceived. I decided to hold off on posting it, because I thought it'd be better to do some technical posts before waxing philosophically. I think it is still appropriate, so as we work on reverse engineering the NAND FTL, here's some food for thought.
Porting Linux to the iPhone is an arduous project. We will be trying to develop an entire suite of device drivers for undocumented hardware and then attempt to run a full-fledged operating system on it. This thread speculates "10 days" or "3 hours" as the amount of time it'd take to get Linux up and running on the iPhone. Perhaps this figure would be accurate on a x86 platform, or other platforms with hardware for which device drivers are already written or for which at least documentation is available, but we have no such luck on the iPhone.
This comment on a O'Reilly Radar article about NerveGas's iPhone Open Application Development book says, with perhaps a little too much vitriol for my taste, that developers should not waste time on the iPhone, a closed platform, and spend time more productively on OpenMoko or Android: truly open platforms. Apple should thus be punished for not making the iPhone open. His point is well-taken though. Reverse engineering Apple's code is inefficient and ought to be unnecessary. Why do I bother when I can just develop on an open platform instead with no such wasted effort?
Finally, I have faced skepticism even from my fellow Dev Team members when I first talked about this project. The iPhone already has a perfectly serviceable operating system that we can develop on. Why does it need another one? Sure, Linux might be cool, but what practical use would it have? How does it justify the tremendous amount of effort that would need to be put in?
So. Why do I bother? Why should we bother?
Part of the answer is that I don't choose which platform I hack on based on how hackable it is. I choose it based on how much I like it. I don't own an OpenMoko device; it simply doesn't look as polished as the iPhone, and support is lacking for it. It wouldn't make sense to buy it to use it, only to buy it to hack on it. While this may work for other people, it's simply not the way a (relatively) starving college student does things. As for the Android, I'm not too convinced about how amazing it will be from the videos I've seen and besides: It doesn't even exist yet! In general, the more people use a device, the more hackers use it, and thus the more it is hacked on. Usability frankly trumps hackability.
The other part of the answer is that iPhone Linux will actually be of tremendous value. There will be no more need to port applications over: The applications already run on the iPhone! Also, with a familiar kernel, we can do all kinds of things I've wanted to do: doing security related work with the wi-fi for example. Plus, knowledge that we are gaining/will have gained about the iPhone hardware will be of incredible practical value to the homebrew iPhone community. We've always wanted to be able to plug in the iPhone as a simple USB mass storage device. With USB and NAND FTL drivers, we can actually implement this ourselves.
Perhaps my most important point is how iPhone Linux will affect the various open platforms in development. The iPhone has revolutionized the way the market thinks about mobile computing and now several mobile platforms are in development: OpenMoko, Google's Android, and Mobile Ubuntu (thought the last is not targeted for phones). All of these projects are based on Linux, and "based on Linux" means that, by definition, they "use the Linux kernel" and the Linux kernel is exactly what we're porting. As long as the kernel works, the rest of the operating system will barely need to be touched at all! (fine print: provided that the working configuration of the kernel can support all the features the userland requires).
Imagine OpenMoko on the iPhone. Android on the iPhone. Ubuntu Mobile on the iPhone. Consumers will have choice, and not some Linux-hippie idealistic choice-for-the-sake-of-choice choice: All of these platforms have major momentum behind them and it is very possible they will end up being better than the iPhone's platform (have better UI, more application support, etc.). Also, imagine what it will mean for the developers of these platforms: A ready userbase of millions of users. If many people can already install and try out one of these platforms, it'll be far easier to attract users to buy the hardware, and developers to develop for the platform. Thus, I do not believe we are harming the open platforms by developing on the iPhone. In fact, if all goes well, we will be allowing them to conquer the Apple iPhone.
Of course, I know the reply to all of this. "That sounds good, now show me the code." It's important not to overpromise and underdeliver, so I will be very cautious. What I have just said is the hope, the best possible outcome. But just having that as a possibility is tantalizing enough to justify working on this project. However, to be honest, my original justification (as stated to the dev team) for working on iPhone Linux was "for Skillz.app", our facetious term for working on something merely to hone one's skill or to satisfy one's curiosity. But honestly, what did you expect from a "hacker"? :)
We have already made more progress with openiboot than many people have anticipated would ever happen. Reverse engineering drivers is a laborious process, but one that doesn't require the luck of finding a security vulnerability: It just happens slowly and steadily, rather than unpredictably. Presumably after the drivers are in place, the Linux kernel will "just work" without too many other changes, since it is designed to be relatively portable, so we ought not to have many problems. After the kernel works, I hope enough developers will become interested and a nice userland can be developed without too much trouble. The userland work is much less risky from a time-investment point of view.
Porting Linux to the iPhone is an arduous project. We will be trying to develop an entire suite of device drivers for undocumented hardware and then attempt to run a full-fledged operating system on it. This thread speculates "10 days" or "3 hours" as the amount of time it'd take to get Linux up and running on the iPhone. Perhaps this figure would be accurate on a x86 platform, or other platforms with hardware for which device drivers are already written or for which at least documentation is available, but we have no such luck on the iPhone.
This comment on a O'Reilly Radar article about NerveGas's iPhone Open Application Development book says, with perhaps a little too much vitriol for my taste, that developers should not waste time on the iPhone, a closed platform, and spend time more productively on OpenMoko or Android: truly open platforms. Apple should thus be punished for not making the iPhone open. His point is well-taken though. Reverse engineering Apple's code is inefficient and ought to be unnecessary. Why do I bother when I can just develop on an open platform instead with no such wasted effort?
Finally, I have faced skepticism even from my fellow Dev Team members when I first talked about this project. The iPhone already has a perfectly serviceable operating system that we can develop on. Why does it need another one? Sure, Linux might be cool, but what practical use would it have? How does it justify the tremendous amount of effort that would need to be put in?
So. Why do I bother? Why should we bother?
Part of the answer is that I don't choose which platform I hack on based on how hackable it is. I choose it based on how much I like it. I don't own an OpenMoko device; it simply doesn't look as polished as the iPhone, and support is lacking for it. It wouldn't make sense to buy it to use it, only to buy it to hack on it. While this may work for other people, it's simply not the way a (relatively) starving college student does things. As for the Android, I'm not too convinced about how amazing it will be from the videos I've seen and besides: It doesn't even exist yet! In general, the more people use a device, the more hackers use it, and thus the more it is hacked on. Usability frankly trumps hackability.
The other part of the answer is that iPhone Linux will actually be of tremendous value. There will be no more need to port applications over: The applications already run on the iPhone! Also, with a familiar kernel, we can do all kinds of things I've wanted to do: doing security related work with the wi-fi for example. Plus, knowledge that we are gaining/will have gained about the iPhone hardware will be of incredible practical value to the homebrew iPhone community. We've always wanted to be able to plug in the iPhone as a simple USB mass storage device. With USB and NAND FTL drivers, we can actually implement this ourselves.
Perhaps my most important point is how iPhone Linux will affect the various open platforms in development. The iPhone has revolutionized the way the market thinks about mobile computing and now several mobile platforms are in development: OpenMoko, Google's Android, and Mobile Ubuntu (thought the last is not targeted for phones). All of these projects are based on Linux, and "based on Linux" means that, by definition, they "use the Linux kernel" and the Linux kernel is exactly what we're porting. As long as the kernel works, the rest of the operating system will barely need to be touched at all! (fine print: provided that the working configuration of the kernel can support all the features the userland requires).
Imagine OpenMoko on the iPhone. Android on the iPhone. Ubuntu Mobile on the iPhone. Consumers will have choice, and not some Linux-hippie idealistic choice-for-the-sake-of-choice choice: All of these platforms have major momentum behind them and it is very possible they will end up being better than the iPhone's platform (have better UI, more application support, etc.). Also, imagine what it will mean for the developers of these platforms: A ready userbase of millions of users. If many people can already install and try out one of these platforms, it'll be far easier to attract users to buy the hardware, and developers to develop for the platform. Thus, I do not believe we are harming the open platforms by developing on the iPhone. In fact, if all goes well, we will be allowing them to conquer the Apple iPhone.
Of course, I know the reply to all of this. "That sounds good, now show me the code." It's important not to overpromise and underdeliver, so I will be very cautious. What I have just said is the hope, the best possible outcome. But just having that as a possibility is tantalizing enough to justify working on this project. However, to be honest, my original justification (as stated to the dev team) for working on iPhone Linux was "for Skillz.app", our facetious term for working on something merely to hone one's skill or to satisfy one's curiosity. But honestly, what did you expect from a "hacker"? :)
We have already made more progress with openiboot than many people have anticipated would ever happen. Reverse engineering drivers is a laborious process, but one that doesn't require the luck of finding a security vulnerability: It just happens slowly and steadily, rather than unpredictably. Presumably after the drivers are in place, the Linux kernel will "just work" without too many other changes, since it is designed to be relatively portable, so we ought not to have many problems. After the kernel works, I hope enough developers will become interested and a nice userland can be developed without too much trouble. The userland work is much less risky from a time-investment point of view.
Thursday, November 6, 2008
Boot Menus, So Far...
So, I'm already getting some boot menus from people! I'll put the entries up here. Please comment and critique them. I will also be giving some of my thoughts:
These two are from chris custren:
These two are from poorlad:
This one is from pH:
That's it so far. If you've got any comments or suggestions about them, leave them here. Here's what came up from a technical/practical perspective: As I've mentioned, gradients are good. It is also very good if you can do things in grayscale (as poorlad did for one of his images, but even the blue scale can be programmatically generated) since it saves on expensive bitmap data. On using the Apple logo, here's what we're willing to do. We're willing to accept a logo with an original, unaltered Apple logo OR a logo created from scratch that resembles the Apple trademark. We cannot accept logos that modifies copyrighted Apple materials. The rationale for using Apple trademarks is because this is merely for identification purposes. If their lawyers have any problems with that, we will respect any requests to take it down.
These two are from chris custren:
These two are from poorlad:
This one is from pH:
That's it so far. If you've got any comments or suggestions about them, leave them here. Here's what came up from a technical/practical perspective: As I've mentioned, gradients are good. It is also very good if you can do things in grayscale (as poorlad did for one of his images, but even the blue scale can be programmatically generated) since it saves on expensive bitmap data. On using the Apple logo, here's what we're willing to do. We're willing to accept a logo with an original, unaltered Apple logo OR a logo created from scratch that resembles the Apple trademark. We cannot accept logos that modifies copyrighted Apple materials. The rationale for using Apple trademarks is because this is merely for identification purposes. If their lawyers have any problems with that, we will respect any requests to take it down.
Wednesday, November 5, 2008
NAND FTL
So the big news yesterday (other than Obama winning the presidency!) is that we have enough of a low-level NAND driver now that we're able to read from NAND! It was epic win. There turns out to be not as much hardware voodoo as, say, Merlot, so that's pretty good news. It seems to work (albeit slowly) and I even wrote the ECC routines today (and those seem to work as well).
Unfortunately, in the course of this, we discovered several unfortunate things. First, I can't seem to find anything that might write to NAND. It's probably not much more complicated and probably reuses a lot of the stuff we've been doing, but it means that we might have to look in the kernel for that code, which sort of bites (a lot of the kernel is in C++ and not as friendly to reverse).
The second thing is the realization that all of Samsung's proprietary FTL code is in this thing. Without being able to understand it, we can't actually map sectors to data and we can't make sense of the NAND data or write new data to it in a useful way. Unfortunately, this code is liable to be ridiculously complex, since it's basically their SDK they ship to everyone. Without it, we can still proceed, but the iPhone can't read Linux's data and Linux can't read iPhone's data. In the worst case, we can't even have both OSes on the NAND at once.
Still, being able to dump NAND through USB is a substantial accomplishment, and we're well on our way.
Unfortunately, in the course of this, we discovered several unfortunate things. First, I can't seem to find anything that might write to NAND. It's probably not much more complicated and probably reuses a lot of the stuff we've been doing, but it means that we might have to look in the kernel for that code, which sort of bites (a lot of the kernel is in C++ and not as friendly to reverse).
The second thing is the realization that all of Samsung's proprietary FTL code is in this thing. Without being able to understand it, we can't actually map sectors to data and we can't make sense of the NAND data or write new data to it in a useful way. Unfortunately, this code is liable to be ridiculously complex, since it's basically their SDK they ship to everyone. Without it, we can still proceed, but the iPhone can't read Linux's data and Linux can't read iPhone's data. In the worst case, we can't even have both OSes on the NAND at once.
Still, being able to dump NAND through USB is a substantial accomplishment, and we're well on our way.
Monday, November 3, 2008
Boot Menu Art
I think the best way to handle this is for anyone interested to submit a proposal via e-mail to me about the boot menu. Then, I can put up the pictures on the blog for people to comment and have an opinion about. We can then figure out which one to use or maybe some combination of proposals or in any case, figure it out in a collaborative community style process.
I don't really have an opinion on what the logo should look like, whether it should be cartoony or not. I only think it should look good. =P
There are very few technical constraints on the boot menu. I'm willing to figure out whatever technical solution there needs be in order to get things to display properly. One thing to keep in mind is that space on the NOR is at a significant premium, and raw pixel data is expensive but procedurally generated stuff like gradients are possible. The only two current menu options are the iPhone OS and the openiboot console, but eventually a Linux option will be added. I can try animation as well, but again, you'd have to have an idea of how it can be done without using up a lot of NOR space.
But yeah, make us pretty!
Some progress on the NAND: Thanks to a huge amount of initial grunt work by CPICH, the NAND project is off the ground. We already have enough to get the proper drive geometries, and I've written and tested the DMA routines that are near the heart of the problem today. Hopefully, we can dump the raw NAND soon and take a peek at what kind of wear leveling data structures we're dealing with.
I don't really have an opinion on what the logo should look like, whether it should be cartoony or not. I only think it should look good. =P
There are very few technical constraints on the boot menu. I'm willing to figure out whatever technical solution there needs be in order to get things to display properly. One thing to keep in mind is that space on the NOR is at a significant premium, and raw pixel data is expensive but procedurally generated stuff like gradients are possible. The only two current menu options are the iPhone OS and the openiboot console, but eventually a Linux option will be added. I can try animation as well, but again, you'd have to have an idea of how it can be done without using up a lot of NOR space.
But yeah, make us pretty!
Some progress on the NAND: Thanks to a huge amount of initial grunt work by CPICH, the NAND project is off the ground. We already have enough to get the proper drive geometries, and I've written and tested the DMA routines that are near the heart of the problem today. Hopefully, we can dump the raw NAND soon and take a peek at what kind of wear leveling data structures we're dealing with.
Sunday, November 2, 2008
Installation, the PMU
While I was waiting for CPICH to finish the first bits of the NAND FTL reverse engineering work, I've been trying to fill in some of the gaps we had in other places, such as the PMU. As promised, there is also now an easy way to install openiboot onto the iPhone. This is great because it will eventually lead to an even leaner and easier QuickPwn in the future.
One of the annoying parts about iBoot in recovery mode is that the thing refuses to charge the iPhone while sitting in recovery mode. The battery just eventually entirely drains. With the new PMU code, openiboot now recharges the battery, so programmers using it (read: me) can just have it sit on the console screen indefinitely. You can also do neat things like check the current battery voltage and check the power supply type the phone is charging from.
The "installation code" consists of porting over my knowledge of reading and modifying img3 files from working on the jailbreaks. I was too lazy to port over the entire xpwn framework, but I wrote up a "diet" version that is sufficient to read and modify img3 files in a limited fashion. img3 files are sort of the new native format of the main part of the NOR (just a bunch of img3 files concatenated together). The upshot is that you can load openiboot as an img3 through iBoot (just like sending an iBEC image) and then type "install" at the console and openiboot will be a permanent stage in your bootloader chain. =P
You can, of course, keep booting up to the iPhone OS as you always do by selecting the option in the boot menu. Installing openiboot isn't very useful except for hackers wanting to hack openiboot.
I also figured out how to parse and modify the NVRAM banks (storing environment variables like "auto-boot", etc.), which was actually pointless complicated (in my opinion). They have two banks consisting of a bunch of partitions with these headers that Apple uses a pointless one-byte custom checksum on. The entire bank is also checksumed with adler32. When NVRAM is modified, the oldest bank is overwritten with the data and becomes the newest bank (which is tracked by an epoch number on each bank). This is so if one bank becomes corrupted, the other can be used as a backup. However, NVRAM hardly contains anything high value so the value of all this trouble is doubtful. Being able to write to NVRAM, though, makes it possible to set auto-boot on and off within openiboot so that we can easily control whether or not to enter iBoot's recovery mode.
Someone asked me how "safe" it was to do the installation, etc. Well, I've been doing it every time I make an update these days, so it's fairly safe. The worst that can happen in the usual case is that you may be forced into a DFU mode restore. Everything will be undone with a restore. Early on, I did have bugs that really screwed things up so that a DFU mode restore was no longer possible, but even that was recoverable. I'll just go over how briefly:
The important thing is to have a backup of the NOR. As I described in a previous posting, it's possible to really screw things up if you erase the SysCfg section of the NOR. If you do that, the iPhone OS will refuse to boot at all since iBoot cannot properly populate the device tree for the kernel. Since restore ramdisks rely on XNU booting, this is Bad News Bears. In addition, the SysCfg section is device specific, so if you do not have a backup, it will be difficult to ever completely recover from erasing it.
Therefore, before you proceed, MAKE A BACKUP OF YOUR NOR. openiboot can do this for you (and subsequently restore your backup if things go wrong).
Load openiboot via loadibec and select the console. Connect with the oibc client. Type in: nor_read 0x09000000 0x0 0x100000
This will read all of NOR into memory. Then type: ~nordump.bin:0x100000
This will transfer the dump over USB onto your computer and save it as nordump.bin.
Supposing you filled the entire NOR with garbage somehow and are unable to boot. You have to get into openiboot to restore the NOR. The problem is that openiboot is only designed to operate in a post-LLB or post-Recovery Mode context, so it cannot be directly booted from DFU mode. Basically, you've got to load a pwned WTF, then a pwned iBSS, and then a pwned iBEC (all of which is available from a custom IPSW). After that, you can use loadibec to load openiboot. Then, you can restore the NOR thus:
!nordump.bin
nor_write 0x09000000 0x0 0x100000
After that, you can reboot and everything should be normal.
Also, I received a few responses for people volunteering to do the art. I'm not sure what the best thing would be, since I don't want anyone putting in effort for nothing, but we do want the best possible results. So, I'll be getting back to you guys about that.
One of the annoying parts about iBoot in recovery mode is that the thing refuses to charge the iPhone while sitting in recovery mode. The battery just eventually entirely drains. With the new PMU code, openiboot now recharges the battery, so programmers using it (read: me) can just have it sit on the console screen indefinitely. You can also do neat things like check the current battery voltage and check the power supply type the phone is charging from.
The "installation code" consists of porting over my knowledge of reading and modifying img3 files from working on the jailbreaks. I was too lazy to port over the entire xpwn framework, but I wrote up a "diet" version that is sufficient to read and modify img3 files in a limited fashion. img3 files are sort of the new native format of the main part of the NOR (just a bunch of img3 files concatenated together). The upshot is that you can load openiboot as an img3 through iBoot (just like sending an iBEC image) and then type "install" at the console and openiboot will be a permanent stage in your bootloader chain. =P
You can, of course, keep booting up to the iPhone OS as you always do by selecting the option in the boot menu. Installing openiboot isn't very useful except for hackers wanting to hack openiboot.
I also figured out how to parse and modify the NVRAM banks (storing environment variables like "auto-boot", etc.), which was actually pointless complicated (in my opinion). They have two banks consisting of a bunch of partitions with these headers that Apple uses a pointless one-byte custom checksum on. The entire bank is also checksumed with adler32. When NVRAM is modified, the oldest bank is overwritten with the data and becomes the newest bank (which is tracked by an epoch number on each bank). This is so if one bank becomes corrupted, the other can be used as a backup. However, NVRAM hardly contains anything high value so the value of all this trouble is doubtful. Being able to write to NVRAM, though, makes it possible to set auto-boot on and off within openiboot so that we can easily control whether or not to enter iBoot's recovery mode.
Someone asked me how "safe" it was to do the installation, etc. Well, I've been doing it every time I make an update these days, so it's fairly safe. The worst that can happen in the usual case is that you may be forced into a DFU mode restore. Everything will be undone with a restore. Early on, I did have bugs that really screwed things up so that a DFU mode restore was no longer possible, but even that was recoverable. I'll just go over how briefly:
The important thing is to have a backup of the NOR. As I described in a previous posting, it's possible to really screw things up if you erase the SysCfg section of the NOR. If you do that, the iPhone OS will refuse to boot at all since iBoot cannot properly populate the device tree for the kernel. Since restore ramdisks rely on XNU booting, this is Bad News Bears. In addition, the SysCfg section is device specific, so if you do not have a backup, it will be difficult to ever completely recover from erasing it.
Therefore, before you proceed, MAKE A BACKUP OF YOUR NOR. openiboot can do this for you (and subsequently restore your backup if things go wrong).
Load openiboot via loadibec and select the console. Connect with the oibc client. Type in: nor_read 0x09000000 0x0 0x100000
This will read all of NOR into memory. Then type: ~nordump.bin:0x100000
This will transfer the dump over USB onto your computer and save it as nordump.bin.
Supposing you filled the entire NOR with garbage somehow and are unable to boot. You have to get into openiboot to restore the NOR. The problem is that openiboot is only designed to operate in a post-LLB or post-Recovery Mode context, so it cannot be directly booted from DFU mode. Basically, you've got to load a pwned WTF, then a pwned iBSS, and then a pwned iBEC (all of which is available from a custom IPSW). After that, you can use loadibec to load openiboot. Then, you can restore the NOR thus:
!nordump.bin
nor_write 0x09000000 0x0 0x100000
After that, you can reboot and everything should be normal.
Also, I received a few responses for people volunteering to do the art. I'm not sure what the best thing would be, since I don't want anyone putting in effort for nothing, but we do want the best possible results. So, I'll be getting back to you guys about that.
Thursday, October 30, 2008
Porting an OS
I've been getting a lot of questions from people that seem to reflect a basic misunderstanding of what it takes to port an operating system onto a new platform. People seem to think that just by writing, say, a boot menu, means that we can stick Android or Windows or whatever onto a device because we can have a menu option for it.
Here's what it takes for an operating system to run on a device:
Because the code cannot interact with the hardware! That is, there are no Linux drivers or Windows Mobile drivers for the hardware that's on the iPhone. We're not even talking about things like the wi-fi won't work or anything silly like that. We're talking about big things, like not being able to start because it doesn't uncompress itself into RAM properly. We're talking about freezing the first time it has to wait for something to happen because it doesn't know how to run the hardware clocks and timers (which is CRITICAL for computers) and doesn't know when to start again.
Thus , if I tried to take some distribution of Linux or Windows or whatever, stick it in memory and start it, absolutely nothing will happen. That's right: nothing. There will be no output because it doesn't know how to run the display, or the USB, or serial. It probably won't even get to the first line of code that tells it to output something because so many things are broken.
So how can we get Linux to boot on the iPhone?
By teaching it how to run the hardware. We take the knowledge gained from getting that boot menu to display and graft it into the Linux kernel. It took an unbelievable amount of devices just to get the boot menu display: clock, timer, vic, mmu, spi, i2c, gpio, system controller, pmu, nor, uart, usb, lcd, buttons. Some of those may seem obvious to you, some work in the background to support the other devices. But all of those had to be reverse engineered and all of them will have to transplanted into the Linux kernel to even get something half-assed booting.
If all of those devices were required to get something as simple as boot menu up, can you imagine what would happen if you tried to boot an operating system that did not know how to run ANY of those devices?
We cannot modify the Windows Mobile kernel because it's closed source, and so there's no way to get it to run on the iPhone.
The critical misunderstanding, I think, is that people think somehow that the OS "sits on top" of the boot menu, and talks to the hardware through the boot menu. Therefore, you can have an "emulation layer" that lets Windows or Linux or whatever talk to the hardware, without having to alter Windows or Linux itself. This is completely false. An operating system, by definition, has direct access to the hardware. Nothing sits between it and the hardware. Once iBoot has loaded the iPhone OS, you can go ahead and wipe it clean from the NOR and the OS will keep running as usual. It's not "running", it's not used or loaded in any way except during the boot process.
The iPhone will never run Windows Mobile directly (virtualization would be possible albeit it would crawl on the iPhone). It will run Linux once we write the drivers for it based on our knowledge of the hardware. Android uses the Linux kernel, though they do modify it to a certain extent. Since the only really hardware dependent parts of an OS is in the kernel, presumably once we install the necessary drivers, Android will run just as well as Linux runs. However, not having even looked at Android's source yet, I really don't have a truly educated opinion at the moment, but let's just say that it's one of this project's primary goals.
Sorry this is so long, but intelligent explanations tend to be long.
P.S. Another question people ask a lot is how long will it take. I can't truly give a good answer to that, because it's sort of dependent on the schedules of the people who work on it, and it also depends on how fast it'll take to write the Linux drivers, and how many unexpected problems crop up. It could go really unexpectedly fast, or we could hit a roadblock. I think outside observers, just reading the commit logs and reading the blog has as much information as I do on how fast things are progressing, so you're free to come up with your own conclusions on how long it will take.
Here's what it takes for an operating system to run on a device:
- The code must be designed for the right CPU. (x86, ARM, PPC)
- The code must be able to interact with the hardware in the way it expects.
Because the code cannot interact with the hardware! That is, there are no Linux drivers or Windows Mobile drivers for the hardware that's on the iPhone. We're not even talking about things like the wi-fi won't work or anything silly like that. We're talking about big things, like not being able to start because it doesn't uncompress itself into RAM properly. We're talking about freezing the first time it has to wait for something to happen because it doesn't know how to run the hardware clocks and timers (which is CRITICAL for computers) and doesn't know when to start again.
Thus , if I tried to take some distribution of Linux or Windows or whatever, stick it in memory and start it, absolutely nothing will happen. That's right: nothing. There will be no output because it doesn't know how to run the display, or the USB, or serial. It probably won't even get to the first line of code that tells it to output something because so many things are broken.
So how can we get Linux to boot on the iPhone?
By teaching it how to run the hardware. We take the knowledge gained from getting that boot menu to display and graft it into the Linux kernel. It took an unbelievable amount of devices just to get the boot menu display: clock, timer, vic, mmu, spi, i2c, gpio, system controller, pmu, nor, uart, usb, lcd, buttons. Some of those may seem obvious to you, some work in the background to support the other devices. But all of those had to be reverse engineered and all of them will have to transplanted into the Linux kernel to even get something half-assed booting.
If all of those devices were required to get something as simple as boot menu up, can you imagine what would happen if you tried to boot an operating system that did not know how to run ANY of those devices?
We cannot modify the Windows Mobile kernel because it's closed source, and so there's no way to get it to run on the iPhone.
The critical misunderstanding, I think, is that people think somehow that the OS "sits on top" of the boot menu, and talks to the hardware through the boot menu. Therefore, you can have an "emulation layer" that lets Windows or Linux or whatever talk to the hardware, without having to alter Windows or Linux itself. This is completely false. An operating system, by definition, has direct access to the hardware. Nothing sits between it and the hardware. Once iBoot has loaded the iPhone OS, you can go ahead and wipe it clean from the NOR and the OS will keep running as usual. It's not "running", it's not used or loaded in any way except during the boot process.
The iPhone will never run Windows Mobile directly (virtualization would be possible albeit it would crawl on the iPhone). It will run Linux once we write the drivers for it based on our knowledge of the hardware. Android uses the Linux kernel, though they do modify it to a certain extent. Since the only really hardware dependent parts of an OS is in the kernel, presumably once we install the necessary drivers, Android will run just as well as Linux runs. However, not having even looked at Android's source yet, I really don't have a truly educated opinion at the moment, but let's just say that it's one of this project's primary goals.
Sorry this is so long, but intelligent explanations tend to be long.
P.S. Another question people ask a lot is how long will it take. I can't truly give a good answer to that, because it's sort of dependent on the schedules of the people who work on it, and it also depends on how fast it'll take to write the Linux drivers, and how many unexpected problems crop up. It could go really unexpectedly fast, or we could hit a roadblock. I think outside observers, just reading the commit logs and reading the blog has as much information as I do on how fast things are progressing, so you're free to come up with your own conclusions on how long it will take.
Wednesday, October 29, 2008
Boot menu done!
Well, that was quick. See, I can actually get things done pretty quickly when it doesn't consisting of banging my head against machine code until it starts making sense. When I actually have the drivers, things like this are easy.
You can use the Hold button to toggle between the menu items (and the option will be highlighted). You can choose the home button to select it. The "openiboot console" option takes you to the command-line interface similar to the one I demonstrated in the last post (you do have to be plugged in via USB and using the openiboot client to talk to it). The "iPhone OS" option chainloads a copy of iBoot stored in NOR under another identifier ('ibot' becomes openiboot and 'ibox' becomes the actual iBoot). I got that set up with a slightly modified version of the QuickPwn ramdisk, but in the future an installer made from a modified version of LogoMe can be run from userland to install openiboot. It's also possible to get openiboot to install openiboot (much like the way GRUB can do it); I'll probably work on that next.
So if anyone likes living on the bleeding edge, they could do that. =P
Most of the hard part was me failing at GIMP putting together the boot menu graphics. I appealed to you blog readers for graphics before, but basically no one responded. Now that there is a working model of what I sort of want, I hope there will be more of a response.
So, please please please redesign the boot menu for me. And possibly come up with a logo for the project we can stick on there. If you're good at this sort of thing, or know someone who is, please put them in touch. This stuff will obviously get a lot of attention in the future and we need nice eye-candy. Thanks!
Boot menu project is a go!
After a huge amount of effort and in-situ experimentation with iBoot (basically a binary massive binary search through the code, disabling some functions to see if I could figure out why my LCD driver wasn't working properly), I managed to get it fully working. The problem was two-fold: first, I forgot to write the first and last bytes of my gamma tables: oops, but easily fixed. The second problem was that apparently iBoot changes the SDIV of the clock in the middle of the initialization process. I'm not even sure yet how many devices the change in clock frequency affects. It certainly affected the LCD, because before there was all sorts of flickering scanline weirdness as one would expect from a misconfigured clock.
Anyway, I reversed the routine that changed the SDIV and implemented it. Seems to work fine now. It's been ages since I looked into the clock speed stuff (pretty much right when I first started this) so I can't say for certain, but I'm pretty sure doing this increases the clock speed (which would make sense).
The LCD driver worked after those fixes and I went onto write a simple framebuffer in a couple of hours, so we can finally get text-mode output on the iPhone screen. It was pretty important to me to get the screen working because even if we can boot a kernel, I wanted the layman to feel like a full-fledged OS was running on the device, and that means display and I/O of some sort.
For a final hurrah, I also wrote some code that lets us detect when the physical buttons (Home, Hold, etc.) were being pressed down. From these pieces, it will be possible to construct a graphical boot menu controlled by those buttons. You could have one option to boot into the iPhone OS, and one option to go into openiboot command-line mode with that text-mode display.
The photo I posted is the current development snapshot running on a first generation iPhone, with oibc (openiboot client) connected and running on my desktop computer. If you have a 2G iPhone or a first-gen iPod touch, you can try it out yourself by checking out the code from Github and compiling it (It's only designed to be built on a Linux machine. You'll be missing some Linux headers otherwise). I wrote some basic notes on how to get it running inside the source tree, but this is not something you're expected to work with unless you're a fairly experience programmer yourself.
Monday, October 27, 2008
openiboot booting!
Well, it's booting. Sort of.
I had some trouble getting the flashed version of it to work because for some reason, 0x0 was not mapped to 0x18000000 when openiboot was loaded. Since all the exception vectors are at 0x18000000, bad ones were being called whenever there was any sort of interrupt. Basically, I just said screw it and rebased the whole program into 0x0. It will basically overwrite whatever exception vector is currently running without worrying about the MMU and such. However, this basically does imply that I don't really understand how the MMU works, so that will have to be fixed.
The end result is what you see above.
The other major roadblock is that the gamma tables remain broken. Even after I chainload iBEC or iBoot over openiboot (as I have done there). The OS boots and everything... just with some really psychadelic colors. =P
So LCD remains a big problem and so does the MMU. But hey, it boots and works (sort of).
Update: LCD now partially fixed. I still need to figure out how to turn the backlight on, but at least chainloaded iPhone OS has normal colors now. =P
LCD driver done?
I had a lot of trouble getting the LCD driver to work. Everything seems to be fine except that when I try to write to the memory address range reserved for the LCD's gamma tables, it doesn't register. It's as if some clock or some device hadn't gotten turned on or something. Therefore, after loading openiboot from iBoot, the screen gets all screwed up.
However, if you load iBEC from iBoot, the screen doesn't get screwed up: you can still use bgcolor and everything works. I thought that meant at first there was something wrong with my LCD init code. I spent a frustrating day carefully auditing it for errors, and I did find two bugs that I fixed, but unfortunately it did not have any effect on the main problem. I got as far as I could with static methods so I decided to perform a series of experiments.
First, I had some trouble chainloading iBoot and iBEC from openiboot. There was a series of fails that I fixed along the way: trouble with USB send (just a silly typo in the client), trouble getting the resulting thing to execute in memory (you've gotta turn off the CPU caches, disable MMU and interrupts for it to work properly. It also can't be run as part of an ISR because, well, iBoot expects to be able to receive interrupts, so I had to move the command processor onto the main thread and just have the ISR queue up commands for the main thread to process). Anyway, those were eventually fixed.
My experiments showed that after openiboot did its inits, chainloaded iBoot and iBEC was unable to reinit the LCD properly (they had the same problem). I narrowed the problem down to the place in power.c where I "turn off" the LCD controller. This happened in the 114 iBoot, so I thought it was necessary. Analyzing the newer 2.x iBoots, that routine was actually removed. Since I am reasonably confident that my syrah_init is functionally identical to their merlot_init and this that power init that when present, causes LCD init to fail in all cases and when absent, allows LCD init to succeed in all cases, I'm pretty sure that's the problem.
So I went ahead and removed it. This may or may not mean I am actually depending on the iBoot that I chainloaded openiboot from for the LCD init. We'll see after I try to replace iBoot entirely in the bootchain.
Anyway, USB is solid as a rock now seemingly and chainloading seems to be working quite well. I'm actually able to load iBoot from NOR, patch it in memory, and then execute it from openiboot. This probably means I'm ready to try flashing the thing again.
Then we'll see how well it truly works.
However, if you load iBEC from iBoot, the screen doesn't get screwed up: you can still use bgcolor and everything works. I thought that meant at first there was something wrong with my LCD init code. I spent a frustrating day carefully auditing it for errors, and I did find two bugs that I fixed, but unfortunately it did not have any effect on the main problem. I got as far as I could with static methods so I decided to perform a series of experiments.
First, I had some trouble chainloading iBoot and iBEC from openiboot. There was a series of fails that I fixed along the way: trouble with USB send (just a silly typo in the client), trouble getting the resulting thing to execute in memory (you've gotta turn off the CPU caches, disable MMU and interrupts for it to work properly. It also can't be run as part of an ISR because, well, iBoot expects to be able to receive interrupts, so I had to move the command processor onto the main thread and just have the ISR queue up commands for the main thread to process). Anyway, those were eventually fixed.
My experiments showed that after openiboot did its inits, chainloaded iBoot and iBEC was unable to reinit the LCD properly (they had the same problem). I narrowed the problem down to the place in power.c where I "turn off" the LCD controller. This happened in the 114 iBoot, so I thought it was necessary. Analyzing the newer 2.x iBoots, that routine was actually removed. Since I am reasonably confident that my syrah_init is functionally identical to their merlot_init and this that power init that when present, causes LCD init to fail in all cases and when absent, allows LCD init to succeed in all cases, I'm pretty sure that's the problem.
So I went ahead and removed it. This may or may not mean I am actually depending on the iBoot that I chainloaded openiboot from for the LCD init. We'll see after I try to replace iBoot entirely in the bootchain.
Anyway, USB is solid as a rock now seemingly and chainloading seems to be working quite well. I'm actually able to load iBoot from NOR, patch it in memory, and then execute it from openiboot. This probably means I'm ready to try flashing the thing again.
Then we'll see how well it truly works.
Wednesday, October 22, 2008
USB fixes
Just a post to indicate things are inching forward slightly. I've been working on debugging USB communications and it seems a lot more stable now. I was basically forced to because my old code only works on computers without usb 2.0, so that ruled out being able to easily work on this project with anything approaching a modern computer. The problem was that I avoided reading the official USB specs (those things are usually overly locutious) and tried to learn instead from sites such as USB in a Nutshell. Unfortunately the driver then failed to properly respond to the device qualifier descriptor which led to epic fail in USB 2.0. The embarrassing thing is iBoot does send this descriptor, but I figured it must be a vendor specific one at the time.
Cmw made me a cable that let's me do serial and USB comm at the same time, which helped a lot in working out the bugs. I'd say it's fairly reliable now; enough for other developers without a serial cable to come in. So how about it, guys?
I've also started to scratch the surface of the NAND driver. Unfortunately, even the lowest level functions are enormously complex. The higher level wear leveling code and data structures even aside. A great deal of it seems to belong to Samsung, since I've found some creepily similar C code lurking around online. Unfortunately, I can't find a complete enough copy of it.
And yes, I'm aware of Android and their source release and yes, I know what you're thinking.
Cmw made me a cable that let's me do serial and USB comm at the same time, which helped a lot in working out the bugs. I'd say it's fairly reliable now; enough for other developers without a serial cable to come in. So how about it, guys?
I've also started to scratch the surface of the NAND driver. Unfortunately, even the lowest level functions are enormously complex. The higher level wear leveling code and data structures even aside. A great deal of it seems to belong to Samsung, since I've found some creepily similar C code lurking around online. Unfortunately, I can't find a complete enough copy of it.
And yes, I'm aware of Android and their source release and yes, I know what you're thinking.
Saturday, August 23, 2008
Similarities and Differences between QuickPwn and ZiPhone
Similarities
Jailbreak
Both utilities jailbreak.
Payload medium
Primary jailbreak payload is placed into iPhone memory for both jailbreaks
Differences
Technique
ZiPhone uses, as the root filesystem device, a pseudo-device that provides a window to an arbitrary section of memory. This memory is not allocated or otherwise reserved by the operating system and hence will be used by other random processes in other random ways and will become more and more corrupted with every CPU clock cycle. The only safe way to use this is to mlock all memory used by the jailbreak binary as soon as possible, and then use data previously uploaded to flash. Anything else will cause either the jailbreak binary to crash at random moments or cause random data to be written to flash. I am not sure why Zibri elected not to implement ZiPhone in a safer fashion.
QuickPwn uses the same mechanism that Apple uses to send its update ramdisk. This memory is both allocated and reserved. It will not crash at random moments, or give you repeating BSD root errors. This is the way the XNU kernel is designed to use ramdisks.
Longevity
ZiPhone hinges on a BUG in iBoot that was quickly fixed by Apple.
QuickPwn uses an iBoot FEATURE that Apple cannot remove without rewriting their own software and undergoing lengthy QA. Even if Apple did change the architecture, it would be straight-forward to simply mimic what they do and adapt to it. The reason QuickPwn can do this is because it relies on a hardware exploit to bootstrap into this phase. Apple cannot fix this problem without changing the manufactured hardware.
Elegance
ZiPhone modifies an existing Apple ramdisk and ships it as a complete set.
QuickPwn contains all-original code and features a very tiny bootstrapper that allows it to use libraries and code that's already on the iPhone.
Not only does ZiPhone's distribution of Apple's binaries violate copyright laws, it also takes up a large portion of room on the ramdisk that could be used for the payload. Keeping its existing algorithm, ZiPhone would never have been able to install Cydia, for example. The maximum feasible ramdisk size is 32 MB; Cydia takes 13 and Apple's library take up a significant amount. With some work, Zibri could possibly make it just under the 32 MB limit, but with the large number of files in Cydia, and the large size of the corruptible area of memory, corruption would be inevitable.
Some history / A personal note
Zibri claims to have "invented the ramdisk jailbreak". Even if this were true, it would have as much relevance to QuickPwn as the 1.0.2 jailbreak does: The techniques used are entirely dissimilar. Not a single step in the process is the same.
However, this is not even true. Before Zibri left, we already had a prototype ramdisk jailbreak in our SVN (which Zibri later leaked parts of). It was written by myself and stored under the very obvious name of "ramdisk-jb" and it contained a modified version of a launchd written by Turbo (who should be considered the father of the ramdisk payload). It basically untarred a SSH installation onto the rootfs. It was rudimentary, and required a lot of work to get up to production standards.
While it's obvious that Zibri has picked every bone of that SVN repository clean, I am puzzled why he did not learn from that example source code. It had mlock and it was written in proper C, unlike the rather make-do replacement of launchd with sh. Perhaps he did not understand the code.
A week before his release, we became aware that Zibri was going to write a ramdisk exploit. We considered racing him to it, but we were constrained by the fact that we had already publicized one working method of jailbreaking: The oft-loathed 1.1.3 soft-jailbreak, which we considered perfectly acceptable until the release of the SDK (we were not aware at the time the SDK release would take so long). In addition, 1.1.3 was a minor update and there was no reason people could not stay on 1.1.2 for awhile longer. The issue is that while a ramdisk jailbreak would certainly be easier and better, we would be burning this great exploit that allowed us to reliably decrypt ramdisks (which we had no other way of doing at the time).
Therefore, we chose not to build our own implementation and instead pursue Pwnage, a longer term project. It was ironic months later that Zibri came to flame us out about releasing the dual-boot method, accusing us of burning the exploit. It was amusing because it was so much lower value than the ramdisk exploit, which he was responsible for burning and really had no future prospects because of pwnagetool.
We are aware that the dual-boot method was the last remaining bit of non-public knowledge from our SVN that he had, and my belief was that the flame was caused by his soreness at losing his last chance at remaining relevant after the pmd ("ramdisk") vulnerability was patched.
Jailbreak
Both utilities jailbreak.
Payload medium
Primary jailbreak payload is placed into iPhone memory for both jailbreaks
Differences
Technique
ZiPhone uses, as the root filesystem device, a pseudo-device that provides a window to an arbitrary section of memory. This memory is not allocated or otherwise reserved by the operating system and hence will be used by other random processes in other random ways and will become more and more corrupted with every CPU clock cycle. The only safe way to use this is to mlock all memory used by the jailbreak binary as soon as possible, and then use data previously uploaded to flash. Anything else will cause either the jailbreak binary to crash at random moments or cause random data to be written to flash. I am not sure why Zibri elected not to implement ZiPhone in a safer fashion.
QuickPwn uses the same mechanism that Apple uses to send its update ramdisk. This memory is both allocated and reserved. It will not crash at random moments, or give you repeating BSD root errors. This is the way the XNU kernel is designed to use ramdisks.
Longevity
ZiPhone hinges on a BUG in iBoot that was quickly fixed by Apple.
QuickPwn uses an iBoot FEATURE that Apple cannot remove without rewriting their own software and undergoing lengthy QA. Even if Apple did change the architecture, it would be straight-forward to simply mimic what they do and adapt to it. The reason QuickPwn can do this is because it relies on a hardware exploit to bootstrap into this phase. Apple cannot fix this problem without changing the manufactured hardware.
Elegance
ZiPhone modifies an existing Apple ramdisk and ships it as a complete set.
QuickPwn contains all-original code and features a very tiny bootstrapper that allows it to use libraries and code that's already on the iPhone.
Not only does ZiPhone's distribution of Apple's binaries violate copyright laws, it also takes up a large portion of room on the ramdisk that could be used for the payload. Keeping its existing algorithm, ZiPhone would never have been able to install Cydia, for example. The maximum feasible ramdisk size is 32 MB; Cydia takes 13 and Apple's library take up a significant amount. With some work, Zibri could possibly make it just under the 32 MB limit, but with the large number of files in Cydia, and the large size of the corruptible area of memory, corruption would be inevitable.
Some history / A personal note
Zibri claims to have "invented the ramdisk jailbreak". Even if this were true, it would have as much relevance to QuickPwn as the 1.0.2 jailbreak does: The techniques used are entirely dissimilar. Not a single step in the process is the same.
However, this is not even true. Before Zibri left, we already had a prototype ramdisk jailbreak in our SVN (which Zibri later leaked parts of). It was written by myself and stored under the very obvious name of "ramdisk-jb" and it contained a modified version of a launchd written by Turbo (who should be considered the father of the ramdisk payload). It basically untarred a SSH installation onto the rootfs. It was rudimentary, and required a lot of work to get up to production standards.
While it's obvious that Zibri has picked every bone of that SVN repository clean, I am puzzled why he did not learn from that example source code. It had mlock and it was written in proper C, unlike the rather make-do replacement of launchd with sh. Perhaps he did not understand the code.
A week before his release, we became aware that Zibri was going to write a ramdisk exploit. We considered racing him to it, but we were constrained by the fact that we had already publicized one working method of jailbreaking: The oft-loathed 1.1.3 soft-jailbreak, which we considered perfectly acceptable until the release of the SDK (we were not aware at the time the SDK release would take so long). In addition, 1.1.3 was a minor update and there was no reason people could not stay on 1.1.2 for awhile longer. The issue is that while a ramdisk jailbreak would certainly be easier and better, we would be burning this great exploit that allowed us to reliably decrypt ramdisks (which we had no other way of doing at the time).
Therefore, we chose not to build our own implementation and instead pursue Pwnage, a longer term project. It was ironic months later that Zibri came to flame us out about releasing the dual-boot method, accusing us of burning the exploit. It was amusing because it was so much lower value than the ramdisk exploit, which he was responsible for burning and really had no future prospects because of pwnagetool.
We are aware that the dual-boot method was the last remaining bit of non-public knowledge from our SVN that he had, and my belief was that the flame was caused by his soreness at losing his last chance at remaining relevant after the pmd ("ramdisk") vulnerability was patched.
Thursday, July 31, 2008
When the bootloader is not the bootloader...
I'm going to address the two comments I received in this post. This basically has nothing to do with Linux, and more to do with iPhone hacking. There's a lot of confusion around with the jailbreak/unlock. The two comments basically hit upon the main points. The main confusion centers around the fact that when you buy an iPhone, you're not just getting a computer, you're getting TWO computers.
What I'm interested in is the S5L8900, the thing that runs the iPhone software. There is another device called the commboard, which has its own processor, nonvolatile memory, boot sequence and everything. It's barely an oversimplification to state that the system board (the S5L8900) and the commboard can only communicate with each other over a serial UART. That is, the only way the system board can control the commboard is with human-readable AT commands! Not very low level at all; they're not very integrated. Being able to hack kernel mode code like iBoot does not give us any more access than we had through minicom on a jailbroken iPhone.
kavkan asked me if iPhone Linux would obviate the unlocks. He then started talking about putting on third-party applications, etc. Putting third party applications on your iPhone is usually referred to as jailbreaking: stuff we do on the S5L8900. When we say unlock, we're usually mean a SIM-unlock. That necessarily means breaking a whole other, entirely distinct, set of security that's on the commboard. A jailbreak makes it easier to do that (because you can now talk to the commboard with that serial UART I discussed earlier), but it's entirely separate.
marc asked me about "bootloader corruption" as it pertains to basebands. As I said earlier, the bootloader I am talking about is on the S5L8900. The baseband/commboard has its own bootloader and its own non-volatile memory (also NOR flash, probably the same bit of flash its bootloader and firmware sits on too). The recovery mechanism on the baseband is far less robust than the one on the S5L8900. The only sure way seems to be using that hardware testpoint to force it to accept a new bootloader, and even that can be defeated by carefully crafting the NOR contents. In other words, it sucks.
In addition, a lot of the problem is due to bad software overwriting the seczone with bad data, stuff that's unique to your phone. Therefore, information is irretrievably lost and there may not be a way to recover.
The disclaimer is, of course, I'm not a baseband expert. This stuff is only what I've surmised by hanging out with some of them. It's kind of funny. On the dev team, w___ and Zf (they're baseband guys) and I were talking about how little we each know about the others' work. We do pretty much the same work, but on different platforms. After I explained what we do on the S5L8900, I think w___ said that he did the same thing "only on the baseband, you have a man sitting on top that does stuff to you for unknown reasons". And for the S5L8900 people, we have a little black box connected to us that either magically works and lets us call people... or not.
What I'm interested in is the S5L8900, the thing that runs the iPhone software. There is another device called the commboard, which has its own processor, nonvolatile memory, boot sequence and everything. It's barely an oversimplification to state that the system board (the S5L8900) and the commboard can only communicate with each other over a serial UART. That is, the only way the system board can control the commboard is with human-readable AT commands! Not very low level at all; they're not very integrated. Being able to hack kernel mode code like iBoot does not give us any more access than we had through minicom on a jailbroken iPhone.
kavkan asked me if iPhone Linux would obviate the unlocks. He then started talking about putting on third-party applications, etc. Putting third party applications on your iPhone is usually referred to as jailbreaking: stuff we do on the S5L8900. When we say unlock, we're usually mean a SIM-unlock. That necessarily means breaking a whole other, entirely distinct, set of security that's on the commboard. A jailbreak makes it easier to do that (because you can now talk to the commboard with that serial UART I discussed earlier), but it's entirely separate.
marc asked me about "bootloader corruption" as it pertains to basebands. As I said earlier, the bootloader I am talking about is on the S5L8900. The baseband/commboard has its own bootloader and its own non-volatile memory (also NOR flash, probably the same bit of flash its bootloader and firmware sits on too). The recovery mechanism on the baseband is far less robust than the one on the S5L8900. The only sure way seems to be using that hardware testpoint to force it to accept a new bootloader, and even that can be defeated by carefully crafting the NOR contents. In other words, it sucks.
In addition, a lot of the problem is due to bad software overwriting the seczone with bad data, stuff that's unique to your phone. Therefore, information is irretrievably lost and there may not be a way to recover.
The disclaimer is, of course, I'm not a baseband expert. This stuff is only what I've surmised by hanging out with some of them. It's kind of funny. On the dev team, w___ and Zf (they're baseband guys) and I were talking about how little we each know about the others' work. We do pretty much the same work, but on different platforms. After I explained what we do on the S5L8900, I think w___ said that he did the same thing "only on the baseband, you have a man sitting on top that does stuff to you for unknown reasons". And for the S5L8900 people, we have a little black box connected to us that either magically works and lets us call people... or not.
Saturday, July 19, 2008
How to fix a bricked iPhone
So how did I manage to FIX the problem I mentioned earlier? The reason I was so vague on the details is that I used a confidential iBoot vulnerability that we didn't want Apple to know even existed! This allowed me to bootstrap openiboot directly from a stock iBSS that was loaded through DFU mode. I still can't tell you exactly what it is, but since geohot already leaked the existence of it, I figure I can tell you it exists and is what I used. :)
Then, it was a simple matter of using openiboot's NOR engine to restore everything. I even can use the new image list parser and AES engine to have a very nice high level interface to the image list, allowing me to "pwn" just with openiboot; no ramdisk futzing around!
The AES code has been in SVN for awhile, but to anyone following jailbreaking news, it's probably obvious why I suddenly, out of the blue, decided to reverse it and write it. Haha. So the night that I committed the AES code, is the night the Dev Team first decrypted the new img3 shit. :)
Then, it was a simple matter of using openiboot's NOR engine to restore everything. I even can use the new image list parser and AES engine to have a very nice high level interface to the image list, allowing me to "pwn" just with openiboot; no ramdisk futzing around!
The AES code has been in SVN for awhile, but to anyone following jailbreaking news, it's probably obvious why I suddenly, out of the blue, decided to reverse it and write it. Haha. So the night that I committed the AES code, is the night the Dev Team first decrypted the new img3 shit. :)
Thursday, July 10, 2008
Conclusive proof that there is no way to (accidentally) brick s5l8900
In the process of testing NOR, I did a pretty lulzy thing. Remember what I said earlier about the memory controller possibly ignoring the first 4 bits? Well, the NOR device ignores the top 12 bits, since it's only 1 MB in total size. This makes a lot of sense. All the designers have to do is basically not wire up some parts of the address bus. So whether you try to address 0x0 or 0x100000 on the NOR, it looks the same to it.
The problem came about because I attemped to add too many images to NOR; a few 140 KB iBoot images can add up pretty quickly. The last one I added ended up shooting into the range reserved for NVRAM (at the end of NOR) and then "wrapping around" to clobber SysCfg, IMG2, and part of the LLB. =P
Hahaha, that's the equivalent of shooting yourself simultaneously in every vital organ. SysCfg stores your SERIAL NUMBER and other unique, irreplaceable pieces of information. The NVRAM contains information iBoot needs to boot up the kernel. The LLB is the thing that securebl tries to load in order to access everything else on NOR and bootstrap iBoot. As the coup de grace, IMG2 contains information that allows the LLB and iBoot to find where the Img2 data starts, so that they can be loaded. This mistake basically was the equivalent of erasing the entire NOR: Every single piece of information on it was rendered unusable. :P
Luckily, as the first test of my NOR driver, I had made a dump of my original NOR, so I was able to restore the SysCfg information. The interesting bit about all this is that you don't even have to do a restore and lose all your data on the NAND even, if you're clever. What I did was let iTunes talk to DFU mode to get into an iBoot. The iPhone actually has a pretty standard DFU mode, as defined by the USB standard. It reports itself as having the correct class, and OpenMoko's dfu-util manages to get, well, something with it. It successfully uploads the iBSS 8900 file (looking at a USB dump, it looks like just the entire file with the 8900 header, signatures, certificates, etc.) but reports that the firmware is corrupted. So at least it seems to use standard status indicators, etc. However, since I couldn't get dfu-util to work, I just used iTunes and pulled the cable out right after it finishes uploading the iBSS. DFU mode doesn't actually change the NOR, it just loads iBSS into memory and executes it. So after this process is done, iBSS will be loaded and you can connect to it via iBooter.
If you had pulled out the cable just a little too late, you can even see the commands iTunes executed on iBSS in the scrollback, Like setpicture and bgcolor. =P
Using the loaded 1.1.4 iBSS, you can bootstrap the necessary actions to restore your NVRAM from backup. I will talk about that in more detail in a future post. But the upshot is, even if you complete kill your "bootloader", and indeed, everything you can possible write to on the iPhone, you can still get things back to normal. :)
Unfortunately, I probably won't have a chance to work on iPhoneLinux stuff much this weekend. I have already been activated by the Dev Team because you-know-what is happening. Time to hax.
The problem came about because I attemped to add too many images to NOR; a few 140 KB iBoot images can add up pretty quickly. The last one I added ended up shooting into the range reserved for NVRAM (at the end of NOR) and then "wrapping around" to clobber SysCfg, IMG2, and part of the LLB. =P
Hahaha, that's the equivalent of shooting yourself simultaneously in every vital organ. SysCfg stores your SERIAL NUMBER and other unique, irreplaceable pieces of information. The NVRAM contains information iBoot needs to boot up the kernel. The LLB is the thing that securebl tries to load in order to access everything else on NOR and bootstrap iBoot. As the coup de grace, IMG2 contains information that allows the LLB and iBoot to find where the Img2 data starts, so that they can be loaded. This mistake basically was the equivalent of erasing the entire NOR: Every single piece of information on it was rendered unusable. :P
Luckily, as the first test of my NOR driver, I had made a dump of my original NOR, so I was able to restore the SysCfg information. The interesting bit about all this is that you don't even have to do a restore and lose all your data on the NAND even, if you're clever. What I did was let iTunes talk to DFU mode to get into an iBoot. The iPhone actually has a pretty standard DFU mode, as defined by the USB standard. It reports itself as having the correct class, and OpenMoko's dfu-util manages to get, well, something with it. It successfully uploads the iBSS 8900 file (looking at a USB dump, it looks like just the entire file with the 8900 header, signatures, certificates, etc.) but reports that the firmware is corrupted. So at least it seems to use standard status indicators, etc. However, since I couldn't get dfu-util to work, I just used iTunes and pulled the cable out right after it finishes uploading the iBSS. DFU mode doesn't actually change the NOR, it just loads iBSS into memory and executes it. So after this process is done, iBSS will be loaded and you can connect to it via iBooter.
If you had pulled out the cable just a little too late, you can even see the commands iTunes executed on iBSS in the scrollback, Like setpicture and bgcolor. =P
Using the loaded 1.1.4 iBSS, you can bootstrap the necessary actions to restore your NVRAM from backup. I will talk about that in more detail in a future post. But the upshot is, even if you complete kill your "bootloader", and indeed, everything you can possible write to on the iPhone, you can still get things back to normal. :)
Unfortunately, I probably won't have a chance to work on iPhoneLinux stuff much this weekend. I have already been activated by the Dev Team because you-know-what is happening. Time to hax.
Wednesday, July 9, 2008
State of iPhone Linux
So the methodology has currently been trying to proceed as quickly as possible, trying to get every device working and aiming for breadth instead of stability. This allows me to do more high-value tasks like reverse engineering, rapidly gaining understanding of the platform instead of just getting bogged down debugging every single thing. Unfortunately, we're paying a bit for it now as I try to get things into gear to put together applications.
First thing is, I don't really trust the current memory structure. For one thing, it's WEIRD. It seems like even if I turn the MMU off, 0x0 is still mapped to 0x18000000. I know the MMU is working, somewhat, because if I allow the heap to run into the place I put my pagetable, bad things happen. =P I understand there's not going to be enough devices or memory to fill out the entire 32-bit address space, though, so maybe there was already some sort of static mapping. I also believe 0x9000000 (the range used by iboot's file transfer facility) is mapped to 0x18100000. That is, 0x0 == 0x80000000 == 0x18000000. The problem is that there are no such mappings in the page table. 0x80000000 to 0x180000000 is set cacheable and bufferable, but is identity mapped. Anyone have enough experience with the hardware to tell me if this makes sense? I mean, maybe it's just that the top 4 bits are just completely ignored by the memory controller.
Second thing is, sometimes I get random freeze-ups and I don't know why. Maybe I'm just hallucinating or screwing up somewhere, or maybe it's just me failing at C (wouldn't be the first time this happened). Anyway, the upshot is, I want to go back through and clean up/refactor the code into its final form. I tried to follow best programming practices as much as possible the first time around, but sometimes it just was too inefficient to do so when dealing with only half-way reverse engineered device drivers.
The third thing is what I'm working on currently. I need openiboot to replace iBoot. I currently have written a pretty simple chainloader. All it does is warm up all the devices as usual, and then load iBoot from NOR and then jumps to it. iBoot is relocateable and should be able to get itself to the right place. Now this works fine from a copy of openiboot that is started from iBoot using "go", but after I flash openiboot onto the "ibot" image in NOR, the device goes straight to DFU. Now either I'm screwing up hardware initialization or there is some additional verification (checksums, probably not signatures) done before LLB wants to load iboot. It may be that the latter is more likely, since I end up in DFU mode rather than a hung device. Not sure if the device is intelligent enough to recognize a failed boot if I don't say, update the powernvram.
After I get this working, the next thing is to see if the gamma table stuff works then (and if not, fix it). After that, the boot menu I talked about can be written. The next thing I want to work on is NAND FTL. That's the last piece before we reach the end of the "openiboot" phase and can move into the Linux phase. Pretty much all the drivers people expect will be ready and the fun can begin.
I know it seems like we're still very far, but I think we've made very concrete and tremendous progress in a fairly reasonable period of time. A lot of things are now clear and the biggest obstacles are not Apple's protections, or a lack of understanding, but merely my own stupid mistakes and typos.
Speaking of horribly stupid mistakes, my next post will be the story of how I almost bricked my phone yesterday night (but not really :P).
First thing is, I don't really trust the current memory structure. For one thing, it's WEIRD. It seems like even if I turn the MMU off, 0x0 is still mapped to 0x18000000. I know the MMU is working, somewhat, because if I allow the heap to run into the place I put my pagetable, bad things happen. =P I understand there's not going to be enough devices or memory to fill out the entire 32-bit address space, though, so maybe there was already some sort of static mapping. I also believe 0x9000000 (the range used by iboot's file transfer facility) is mapped to 0x18100000. That is, 0x0 == 0x80000000 == 0x18000000. The problem is that there are no such mappings in the page table. 0x80000000 to 0x180000000 is set cacheable and bufferable, but is identity mapped. Anyone have enough experience with the hardware to tell me if this makes sense? I mean, maybe it's just that the top 4 bits are just completely ignored by the memory controller.
Second thing is, sometimes I get random freeze-ups and I don't know why. Maybe I'm just hallucinating or screwing up somewhere, or maybe it's just me failing at C (wouldn't be the first time this happened). Anyway, the upshot is, I want to go back through and clean up/refactor the code into its final form. I tried to follow best programming practices as much as possible the first time around, but sometimes it just was too inefficient to do so when dealing with only half-way reverse engineered device drivers.
The third thing is what I'm working on currently. I need openiboot to replace iBoot. I currently have written a pretty simple chainloader. All it does is warm up all the devices as usual, and then load iBoot from NOR and then jumps to it. iBoot is relocateable and should be able to get itself to the right place. Now this works fine from a copy of openiboot that is started from iBoot using "go", but after I flash openiboot onto the "ibot" image in NOR, the device goes straight to DFU. Now either I'm screwing up hardware initialization or there is some additional verification (checksums, probably not signatures) done before LLB wants to load iboot. It may be that the latter is more likely, since I end up in DFU mode rather than a hung device. Not sure if the device is intelligent enough to recognize a failed boot if I don't say, update the powernvram.
After I get this working, the next thing is to see if the gamma table stuff works then (and if not, fix it). After that, the boot menu I talked about can be written. The next thing I want to work on is NAND FTL. That's the last piece before we reach the end of the "openiboot" phase and can move into the Linux phase. Pretty much all the drivers people expect will be ready and the fun can begin.
I know it seems like we're still very far, but I think we've made very concrete and tremendous progress in a fairly reasonable period of time. A lot of things are now clear and the biggest obstacles are not Apple's protections, or a lack of understanding, but merely my own stupid mistakes and typos.
Speaking of horribly stupid mistakes, my next post will be the story of how I almost bricked my phone yesterday night (but not really :P).
Friday, July 4, 2008
LCD working! (sort of...)
After many frustrating hours debugging all the pieces that fit into this LCD driver, it now behaves as expected. Unfortunately, I still can't get the gamma table to install from a clean start-up (where openiboot powers down everything that iboot powers down before starting them back up). I think this is probably because the gamma table installation code that I reversed from iBoot is only designed to work in situations where the code is directly loaded from LLB. The other possibility is that I made a mistake somewhere in merlot_init. I know that the rest of my code is good because it generates the exact same register configuration that iBoot does. merlot_init is more difficult to diagnose because of all the SPI commands flying around. Most of them just read and write to registers on the LCD panel, though.
What's interesting about the gamma table installation function in iBoot is that it uses a custom, rudimentary form of compression. The gamma table is just a mapping from every R, G and B value (0 - 0xFF) to a range from 0 - 0x3FF. There are three table, one for red, one for green and one for blue. Each table has 0x100 4-byte values. That adds up to 3 kilobytes worth of tables! Since space on the NOR is at a premium, it's sensible that Apple use some kind of fast compression to store the gamma table data. Their compression encodes values two bits at a time, and takes advantage of the fact that each entry in the gamma table will be fairly close to the value of the last entry, just offset a little. They use the two bit control codes to select how to manipulate the differences. I‘m not sure if this is any sort of standard compression, but I thought it was amusing that they had this in here. You can look in the iPhone Linux SVN for the code.
Still, we have working code and just a relatively minor problem! I think that means we could probably start some sort of logo contest. Here's how I think it will work: We need art for the bootloader menu interface. The art will naturally have to feature iPhone Linux, so that means that we also require a logo for this project. So we need the following things: A logo for iPhone Linux, and a logo (could be just a variant of the iPhone Linux logo) for openiboot (not sure what the canonical capitalization of that should be; we will probably use the logo as the reference). We also need art for the bootloader based on those logos. What I'm envisioning is just the logos, bracketed by arrows to make it obvious that you can select between them. Then, the standby key can be used to toggle through the various choices, and the home key will boot the selected one.
The winning set of artwork will be selected by community consensus. The community is very small as of yet, so I'm sure it'll be easy to just to talk it over with everyone and decide what people like. It's important to note that even if your logo is chosen, if someone offers us something prettier, we will switch, so don't do this if your feelings are easily hurt. =P
I really appreciate what artists do and the care and craftsmanship that they put into their work. You guys have a skill that I will never have and I know that it is a lot to ask to have you put that kind of work into something that might not even be used. However, I and many others have put analogous care into crafting the code for this project, and that is our salute to the community. So let's make something beautiful together.
To submit an entry, just contact me or cmw with it in some way, shape or form. Either on my gmail account (take a wild guess what my gmail address is :P), or on IRC, or here in the comments, or whatever. We'll see to it that everyone gets a chance to take a look at it.
What's interesting about the gamma table installation function in iBoot is that it uses a custom, rudimentary form of compression. The gamma table is just a mapping from every R, G and B value (0 - 0xFF) to a range from 0 - 0x3FF. There are three table, one for red, one for green and one for blue. Each table has 0x100 4-byte values. That adds up to 3 kilobytes worth of tables! Since space on the NOR is at a premium, it's sensible that Apple use some kind of fast compression to store the gamma table data. Their compression encodes values two bits at a time, and takes advantage of the fact that each entry in the gamma table will be fairly close to the value of the last entry, just offset a little. They use the two bit control codes to select how to manipulate the differences. I‘m not sure if this is any sort of standard compression, but I thought it was amusing that they had this in here. You can look in the iPhone Linux SVN for the code.
Still, we have working code and just a relatively minor problem! I think that means we could probably start some sort of logo contest. Here's how I think it will work: We need art for the bootloader menu interface. The art will naturally have to feature iPhone Linux, so that means that we also require a logo for this project. So we need the following things: A logo for iPhone Linux, and a logo (could be just a variant of the iPhone Linux logo) for openiboot (not sure what the canonical capitalization of that should be; we will probably use the logo as the reference). We also need art for the bootloader based on those logos. What I'm envisioning is just the logos, bracketed by arrows to make it obvious that you can select between them. Then, the standby key can be used to toggle through the various choices, and the home key will boot the selected one.
The winning set of artwork will be selected by community consensus. The community is very small as of yet, so I'm sure it'll be easy to just to talk it over with everyone and decide what people like. It's important to note that even if your logo is chosen, if someone offers us something prettier, we will switch, so don't do this if your feelings are easily hurt. =P
I really appreciate what artists do and the care and craftsmanship that they put into their work. You guys have a skill that I will never have and I know that it is a lot to ask to have you put that kind of work into something that might not even be used. However, I and many others have put analogous care into crafting the code for this project, and that is our salute to the community. So let's make something beautiful together.
To submit an entry, just contact me or cmw with it in some way, shape or form. Either on my gmail account (take a wild guess what my gmail address is :P), or on IRC, or here in the comments, or whatever. We'll see to it that everyone gets a chance to take a look at it.
Thursday, July 3, 2008
LCD status
Looks like the LCD initialization stuff wasn't as simple as I thought. Depending on the way you look at it, there are three or four major initialization steps for the display. The first step initializes the display controller chip, sets the clock and everything. The could of messages you see about the framerate and clock are from this step. The second step ought to initialize the framebuffer for the display controller. I've reverse engineered and implemented those two steps so far.
The third step is to communicate with the LCD panel itself, and likely configure it and configure the display controller for it as well. This is the infamous "merlot_init" function. I have no idea what merlot is (other than a wine variety). Could be the codename for the driver, or the display controller. Can't be for the LCD itself, since its design to hande many different panel types. It's a pretty funny name, though, so I've started working on syrah_init. Syrah, because the 2005 Dalla Vina vintage was an award-winner at the Spring Beer & Wine Fest that I went to last year.
The problem with merlot_init is that it uses GPIO, SPI, I²C as well as memory mapped registers to communicate with the panel and/or the display controller. That's almost every single bus on the iPhone, so basically I had to write drivers for those controllers as well before I could start on merlot_init. Those are now written; no idea if they work, but they're written. It's going to be a big pain in the butt to debug such a complicated driver. There's just too many things that can go wrong.
It's possible I might start implementing some of the PCF50633 (iPhone's power management unit) functionality. I can at least access powernvram (the general purpose memory registers on the PMU, really) pretty simply using I²C and I ought to be able to test that piece out anyway.
On another note, for the longest time I couldn't figure out what those gpmem registers did on the PMU. I reversed some code that manipulated them, but I couldn't figure out where the data was being used, etc. However, I didn't spend much time on it because I didn't think they were going to be very important for what I was doing (since their values don't affect the initialization of any of the other drivers). Well, last night I told MuscleNerd about this for some reason, and he pointed out that there's a command called "powernvram" in iBoot that, no less, attaches descriptions to every single one of those registers! They just keep track of boot failures and stupid stuff like that. It just goes to show that you can't spend TOO much time just doing static RCE. Sometimes you've got to fire up the actual application, or at least give the old ztringz a go. ;)
The third step is to communicate with the LCD panel itself, and likely configure it and configure the display controller for it as well. This is the infamous "merlot_init" function. I have no idea what merlot is (other than a wine variety). Could be the codename for the driver, or the display controller. Can't be for the LCD itself, since its design to hande many different panel types. It's a pretty funny name, though, so I've started working on syrah_init. Syrah, because the 2005 Dalla Vina vintage was an award-winner at the Spring Beer & Wine Fest that I went to last year.
The problem with merlot_init is that it uses GPIO, SPI, I²C as well as memory mapped registers to communicate with the panel and/or the display controller. That's almost every single bus on the iPhone, so basically I had to write drivers for those controllers as well before I could start on merlot_init. Those are now written; no idea if they work, but they're written. It's going to be a big pain in the butt to debug such a complicated driver. There's just too many things that can go wrong.
It's possible I might start implementing some of the PCF50633 (iPhone's power management unit) functionality. I can at least access powernvram (the general purpose memory registers on the PMU, really) pretty simply using I²C and I ought to be able to test that piece out anyway.
On another note, for the longest time I couldn't figure out what those gpmem registers did on the PMU. I reversed some code that manipulated them, but I couldn't figure out where the data was being used, etc. However, I didn't spend much time on it because I didn't think they were going to be very important for what I was doing (since their values don't affect the initialization of any of the other drivers). Well, last night I told MuscleNerd about this for some reason, and he pointed out that there's a command called "powernvram" in iBoot that, no less, attaches descriptions to every single one of those registers! They just keep track of boot failures and stupid stuff like that. It just goes to show that you can't spend TOO much time just doing static RCE. Sometimes you've got to fire up the actual application, or at least give the old ztringz a go. ;)
Friday, June 27, 2008
LCD Driver
I lied, this is not the "Why iPhone Linux?" post; it's boring and philosophical anyway. Instead, I've got some hacking news!
So I've been putting off the LCD driver because I always thought it would be big and complicated. A long time ago, before I looked at all the USB code, I took one look at merlot_init and got scared off, but upon revisiting it, it really doesn't look that bad. I also was afraid of it possibly needing complicated interrupt routines, but now I can see that it only uses interrupts to dispatch spi and possibly gpio i/o, which is generic and not too bad.
I also was wondering why openiboot, as it is, makes the screen fade to (nearly) white in what pytey has called "disco lights" behavior. openiboot is obviously breaking the already initialized LCD driver somewhere. At first I thought it was because iBoot has stopped responding to interrupts necessary for the upkeep of the LCD, but an infinite loop before anything has happened will cause the LCD to remain the same. Therefore, I put a bunch of rudimentary "breakpoints" (infinite while-loops really) around the beginning of openiboot, and discovered that the LCD breaks when I initialize the system controller's power feature, which just turns off most devices. If I don't do that, the LCD stays initialized. What's more, I can just write raw ARGB (well, from the perspective that I am writing little-endian integers that are 0xAARRGGBB, it's actually BGRA in memory) into the vram range (as labeled here) and it'll show up on screen! This is without any sort of interrupt support or depending on any other memory resident code. So apparently, all that is needed is the initialization routine for basic functionality!
This is great news, and the upshot is that I think I can finish the LCD driver by the end of this weekend. In addition, cmw has already written a framebuffer that can be used after driver init, so we can display text and whatever we want.
So I've been putting off the LCD driver because I always thought it would be big and complicated. A long time ago, before I looked at all the USB code, I took one look at merlot_init and got scared off, but upon revisiting it, it really doesn't look that bad. I also was afraid of it possibly needing complicated interrupt routines, but now I can see that it only uses interrupts to dispatch spi and possibly gpio i/o, which is generic and not too bad.
I also was wondering why openiboot, as it is, makes the screen fade to (nearly) white in what pytey has called "disco lights" behavior. openiboot is obviously breaking the already initialized LCD driver somewhere. At first I thought it was because iBoot has stopped responding to interrupts necessary for the upkeep of the LCD, but an infinite loop before anything has happened will cause the LCD to remain the same. Therefore, I put a bunch of rudimentary "breakpoints" (infinite while-loops really) around the beginning of openiboot, and discovered that the LCD breaks when I initialize the system controller's power feature, which just turns off most devices. If I don't do that, the LCD stays initialized. What's more, I can just write raw ARGB (well, from the perspective that I am writing little-endian integers that are 0xAARRGGBB, it's actually BGRA in memory) into the vram range (as labeled here) and it'll show up on screen! This is without any sort of interrupt support or depending on any other memory resident code. So apparently, all that is needed is the initialization routine for basic functionality!
This is great news, and the upshot is that I think I can finish the LCD driver by the end of this weekend. In addition, cmw has already written a framebuffer that can be used after driver init, so we can display text and whatever we want.
Tuesday, June 24, 2008
An introduction to the blog
Sometimes, I like to write down my thoughts about stuff. My graphomania tends to be sporadic, but it is nice to have a place where I can exercise it when I experience it. The dev team has a bit of a taboo against blogging, which is why I haven't done it before. "Code, not ego" is our creed, and some of us believe that blogging sometimes lead to the interests of the community or the team being compromised in order to gain attention. However, I don't see anything wrong with sharing some of my love for the art... as long as I practice discretion and good taste.
Besides, we have an "official blog" now, so I feel like the playing field is open. ;)
This blog is for the developers of iPhone Linux. Any developer who's working on the project (sadly, a very short list of individuals) can ask me for write permissions here. The format will be kept largely informal. Major announcements and technical information will go on the wiki, but I'll personally be dumping a lot of technical information here on the blog as well when I can't decide/can't be bothered to condense and format it for the wiki. Views represented here are of the individual developer and not of the project in general.
Let me introduce myself though: Hi, I'm planetbeing. I'm currently, I suppose, the lead developer on the iPhone Linux project by virtue of circumstance. I was introduced to this project by cmw, a hacker who has a ton of experience porting Linux and other operating systems to a variety of equipment in the past. Unfortunately, he's currently largely burdened with other commitments. Even so, he helps out as much as he can, and he has some very exciting projects bubbling in the background that I'm sure you'll hear about soon. :)
I'm also a member of the iPhone Dev Team. While iPhone Linux has not official or direct connection with the Dev Team (I just happen to be on the Dev Team and also working on this project), its members often lend me valuable insight into many, many problems. Remember, less than a year ago I hadn't reversed a single line of code. I brought to the table (I hope) a quick and ready mind but only by hanging out with them was I able to fill it so fast.
Lastly, I tend to be overly locquacious in writing. Honestly, most people have one paragraph introduction posts and this is already War and Peace. For that, I apologize and I hope you will stick with me nevertheless.
My next post will be entitled "Why iPhone Linux?" It's something I often am challenged on and I thought I would write up an article justifying all the time I'm spending on it. :)
Besides, we have an "official blog" now, so I feel like the playing field is open. ;)
This blog is for the developers of iPhone Linux. Any developer who's working on the project (sadly, a very short list of individuals) can ask me for write permissions here. The format will be kept largely informal. Major announcements and technical information will go on the wiki, but I'll personally be dumping a lot of technical information here on the blog as well when I can't decide/can't be bothered to condense and format it for the wiki. Views represented here are of the individual developer and not of the project in general.
Let me introduce myself though: Hi, I'm planetbeing. I'm currently, I suppose, the lead developer on the iPhone Linux project by virtue of circumstance. I was introduced to this project by cmw, a hacker who has a ton of experience porting Linux and other operating systems to a variety of equipment in the past. Unfortunately, he's currently largely burdened with other commitments. Even so, he helps out as much as he can, and he has some very exciting projects bubbling in the background that I'm sure you'll hear about soon. :)
I'm also a member of the iPhone Dev Team. While iPhone Linux has not official or direct connection with the Dev Team (I just happen to be on the Dev Team and also working on this project), its members often lend me valuable insight into many, many problems. Remember, less than a year ago I hadn't reversed a single line of code. I brought to the table (I hope) a quick and ready mind but only by hanging out with them was I able to fill it so fast.
Lastly, I tend to be overly locquacious in writing. Honestly, most people have one paragraph introduction posts and this is already War and Peace. For that, I apologize and I hope you will stick with me nevertheless.
My next post will be entitled "Why iPhone Linux?" It's something I often am challenged on and I thought I would write up an article justifying all the time I'm spending on it. :)
Subscribe to:
Posts (Atom)