What is a memory smasher?

A memory smasher is the name for what happens when valid memory is replaced with garbage. When memory is replaced with garbage this does not always cause an immediate problem. Often the memory that is replaced with garbage is not used for some time and the garbage memory acts as a booby trap waiting for the program to use it and crash.

What causes crashes?

One of the most common causes of crashes is when bookkeeping memory is replaced with garbage. In addition to the objects you create and use directly there is a large amount of memory used to keep track of function call state, memory bookkeeping, dynamic method dispatch tables, or other runtime features of the programing language. When bookkeeping memory is read and then used it it can result in completely bogus instructions like freeing memory that does not exist or trying to call a function that does not exist. Those invalid instructions are what cause the crash.  One of the most common crash points is in memory management functions like (m)alloc or free.

Where does it crash?

What makes memory smasher crashes difficult to debug is the crash point has nothing to do with the source of the problem. A single line/function call could corrupt memory, but the crash could happen in a completely unrelated codebase. The program could then run for seconds, minutes, or hours before the garbage memory causes a problem. The crash points will vary from crash to crash because it is not predictable, what memory was replaced with garbage,  what is in the garbage, or when the garbage will be used. Usually the crashes aggregate around wherever your program is spending most if its time. Each time your program touches memory, especially the memory used for internal bookkeeping, it has a chance of touching the garbage memory and crashing. If every line of your program had a .01% chance of crashing you would see a lot of crashes on the lines you call most frequently.

How do I debug it?

The best way to debug memory smashers is to run unit tests with Address Sanitizer (ASAN). ASAN will add safeguards around writing memory and help detect incorrect memory writing.

Some tricks with pip install

TL;DR Don't user sudo python setup.py intall use pip install -e ./ --user

I regularly Go through the following process:

1. Python packages are confusing
2. I should read the docs
3. pip is cool
4. I have no idea what I am doing

But this time I actually learned something useful.

Background 1: easy_install and pip

If you have spent any reasonable amount of time you have probably learned about easy_install and pip. You have probably also generally learned that easy_install is not good and should be replaced with pip in most cases.

Background 2: Working with Package Source

So pip install works fabulously with packages in PyPI, but that does not help you if you work with actual package source. If you deal with the source for python package source there are three common ways to deal with developing and testing on the package source itself.

1. Work in the same directory as the package source
2. Modify sys.path to add your working directory
3. Install the package using sudo python ./setup.py install

One and Two are just fragile, crazy, and if you have ever tried it you know somewhere in the back of your head this is bad. Three is theoretically the right way to install and test python Packages, but it has a number of downsides.

1. Requires sudo
2. Leaves a build directory owned by root
3. Needs to be re-run every time the source changes
4. Can't easily uninstall packages installed this way

Useful Thing 1: Using pip for local source installation

sudo pip install ./ can be used instead of setup.py. The solves the second and fourth problem on our list. No more build directory is left behind after the installation and sudo pip uninstall <Package> will uninstall the package in one command

Useful Thing 2: Installing in non-System paths

Adding a --user flag to pip install puts the installed package at ~/Library instead of /Library. Because ~/Library is owned by the user and not root there is no need for sudo.

Useful Thing 3: Installing Package symlinks*

By adding the -e flag to pip a symlink to the source source is installed instead of the byte-code compiled source. It is not really a symlink, but it is a close enough analogy to understand what is happening.

Summary

Developing and Testing Packages locally can be greatly improved by using pip install -e ./ --user. This keeps the "installed" package up to date with the source and can be easy uninstalled with pip uninstall <Package>

100 Reasons to love python

I finally got around to editing the examples from my 100 Reasons why I love python talk. Feel free to ask questions if anything is unclear. Some of the examples are my own sparse presenters notes instead of fully fleshed out examples.

Lessons From iOS: NSURLConnection

Edit:  NSURLConnection is dead. long live  NSURLSession

I am working on an app for my University, LeTourneau University. The app pulls data from a bunch of web services.  Sometimes these services have nice XML/JSON/RSS interfaces. Sometimes you have to result to good old fashion HTML scrapping. The trick will be how to download the content of a HTML document on iOS.

Usually when you work with HTML you are displaying it in a web view of some flavor. But when you are scrapping HTML you want the content of the HTML doc itself. Browsing around the iOS docs will lead you to some potentially useful classes:

• NSURL
• NSURLRequest
• NSURLConnection
• NSURLResponse

The NSURL class is straightforward enough. It stores a manipulates URLs. The NSURLRequest is representation of request to load a URL. The NSURLConnection is a way to execute a URL request. The NSURLResponse is the results of the executed request. 

The real key is the NSURLConnection and the various NSURLConnection delegates. NSURLConnectionDelegate, NSURLConnectionDownloadDelegate, and NSURLConnectionDataDelegate are complementary delegates designed to configure a specific NSURLConnection. These delegates have been extensively adjusted, modified, and overhauled. Most of the methods in these delegates have become obsolete since iOS 5 (more on this later).  The first step is to fetching data is to to set up the NSURLConnection. NSURLConnections can be performed synchronously or asynchronously. To prevent blocking the main thread I recommend sending requests asynchronously.

-(void)performRequest
{

    NSURL * url =[[NSURL alloc] initWithString:@"http://myurl.com"];
    NSURLRequest * request = [[NSURLRequest alloc] initWithURL:url];
    NSURLConnection * connection =[[NSURLConnection alloc]initWithRequest:request 
																 delegate:self];
    
}

Once the request  has been performed all the rest of the processing occurs in the delegate methods. If your URL is secure then you will need to implement the following two methods to ensure access.

- (void)connection:(NSURLConnection *)connection willSendRequestForAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge
{
    NSUserDefaults *userDefaults = [NSUserDefaults standardUserDefaults];
    if ([challenge previousFailureCount] == 0)
    {
        NSURLCredential * cred = [NSURLCredential credentialWithUser: @"username"
                                                            password: @"password"
                                                         persistence: NSURLCredentialPersistenceForSession];
        [[challenge sender] useCredential:cred forAuthenticationChallenge:challenge];
    }
    else
    {
        //You don't have authentication
    }
}

-(BOOL)connectionShouldUseCredentialStorage:(NSURLConnection *)connection
{
    return YES;
}

The first method is how to provide authentication for the URL. The second method is to use authentication credentials already stored either in the keychain or earlier in your applications code. Once you started the connection and provided any authentication required you should check for connection failures.

-(void)connection:(NSURLConnection *)connection didFailWithError:(NSError *)error
{
    //handle errors
}

The final step is to receive the data and process it.

-(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
    NSString * html = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
    NSLog(@"%@",html);
}

The didReceiveData method is part of an informal protocol called NSURLConnectionDataDelegate. All of the methods in the NSURLConnection delegate related to receiving data are deprecated. You have three options that aren't deprecated.The NSURLConnectionDataDelegate method shown above. The NSURConnection class provides

+ (void)sendAsynchronousRequest:(NSURLRequest *)request queue:(NSOperationQueue *)queue completionHandler:(void (^)(NSURLResponse*, NSData*, NSError*))handler;

But this method does not have a delegate option for handling authentication. This option only works for unsecured URL's. The last option is to use the NSURLConnectionDownload delegate. This downloads the URL's contents to a temporary file that you can then copy. I tried to get this to work but had a headache trying to debug the errors. Edit: but in the mean time Apple has added the fabulous new NSURLSession class and a whole host of new API's I highly encourage you to check them out.