Attempting to send xx_ids: []

I have been recently working on an REST API (with rails 4.x) and a client wanted to send an empty array to indicate truncation of an association. The resource already existed and she wanted to send a PUT (with all attributes) including an empty array for a related collection that should now be empty. This could be the case if an author of a blog posts decides to delete all its comments and sends comments: []. From a REST perspective I found this reasonable, and we went for adding myassoc_ids: [] to the relevant controller. Everything looked fine and we deployed the code to staging (which runs in a production environment)

Next, she complained that the API requests fail with an 422 error. I took a look at the logs and found an entry that strong_parameters was complaining about:

found unpermitted parameters: myassoc_ids

This was really confusing since it had been added to the controller. Re-checking the log we found the cause.

Value for params[:object][:myassoc_ids] was set to nil, because it was one of [], [null] or [null, null, …]. Go to http://guides.rubyonrails.org/security.html#unsafe-query-generation for more information.

I took a look at the section of the documentation and it says that empty arrays are turned into nil to make sure, the developer does not perform a nil check as a sensitization which is not sufficient for the example cases in the warning.

However the fact that rails turns empty arrays into nil, totally removes the ability to properly accept empty collections to indicate truncation of the collection. It turns out, that other people have already discusses this behavior and the rails team added this flag: config.action_dispatch.perform_deep_munge. Just as a note: deep_munge is the action that performs this parameter conversion. So everything is fine, just add a small test and disable deep_munge via config and this should be fixed quickly.

…

Well it should be, but some hours later I found myself failing to reproduce the problem in my test-suite. I am using rspec and I wrote a request spec to make sure all middleware parts and parameter interactions happen as they do in reality. However when I looked at the parameters in my controller they were always missing the empty associations. So after trying over and over again, I decided to run a debugger, since I had a certain feeling this might be a problem in rspec or in rails’ integration tests, since rails also added deep_munge in the first place.

After stepping deep down through rspec and action_dispatch into rack-test utils I finally found some methods that build a rack input from the given request.

There is a method called : build_nested_query in rack-test/lib-rack/test/utils.rb that handles arrays in a typecase by using each. Thus the processing of the request does not do anything on an empty array.

After searching at github I found This bug, which is exactly the bug I hit. Unfortunately the issue is more than a year old and it appears as the maintainer is not working on fixing it.

Now I have to change the configuration without writing a test to prevent regression. Hopefully there will be some feedback for this bug soon.

I skipped through everything quiet fast, so lets try to understand what happened in detail and how the different oddities and bugs worked together.

Understanding the problem

So what happened in detail and how did the different parts interact with one another?

StrongParameter per environment configuration

We only noticed the problem in staging (production) environment, since strong_parameters usually is configured to raise in production only and not in development or testing. Testing would require some heavy refactoring since usually attributes_for or similar is used for the controller and request specs and thus the parameter validation would fail. However I think since this is a fork of logic, every user of strong parameters, including myself, should check how to DRYly make the permitted parameters available for the controller and the tests. One suggestions I though of is, that the parameter specification could be moved into a dedicated object. I talked to somebody in the freenode #ruby channel and he told me they do something similar by using pundit policies for the parameter validation, which rally sounds interesting. I think I might write a followup on this topic.

So I just recommend trying to configure strong_parameters the same way for all environments.

Strong Parameter types

Per default strong parameters only permits scalar types as described in the readme. To allow an array, like I needed to, I had to set it explicitly to myassoc_ids: []

deep_munge

However since deep_munge converts arrays into nil, the type of the permitted value does no longer match and thus the resulting error is

unpermitted parameter …

even though the parameter is permitted, but due to deep_munge’s interference, the type does no longer match. Thus the error got really confusing at let us search at the wrong place for quiet some time.

It seems as If deep_munge is gone in the latest 5.x versions of rails and the security fixes have been moved into ActiveRecord. I am not sure if that is the right way, however at least sending empty arrays will work.

TDD and rack-test

After we understood all those interactions it was bad luck, that this case is also not supported by rack-test at all. This means, that nobody is able to write a test, that verifies the behavior of sending an empty array as parameter.

I wonder if this is possible? Is there really NOBODY sending empty collections to indicate an empty set? If somebody has an idea for a better way to do this, please let me know.

However merging the change into master feels uncomfortable without having regressions tests added to the project.

Hopefully the rack-test maintainer will respond to the open issues and pull requests or respond to my email, in which i offered supporting him, in case he has no time for working on the project anymore.

Wrap up

I don’t want to complain too much about decisions the rails team made for security concerns, or that open source software like rack-test has bugs. I think maybe there is just too much magic and indirection in this basic rails-api stack I was working on, or maybe it really just was bad luck.

Hopefully sharing this experience helps someone else finding the solution with less effort, in case she enters the same situation.