SIM/DPM relay response corrupts non-latin utf8 characters

[jldec]

We are using a SIM/DPM form which collects payment/order information and POSTs the order details directly from customers browsers to the Authorize.Net gateway.

When users enter text using characters that are not in iso-8859-1, like the cyrillic и (U+0438), these characters are correctly processed by the gateway (they show up properly in the customer's receipt email) but they appear to arrive incorrectly in the POST to the relay response URL.

E.g the cyrillic и (U+0438) is encoded as %38 in the relay response POST (which renders as the character 8)

The problem can be verified by entering say ииииииии into the x_first_name field and then pointing x_relay_url to https://developer.authorize.net/tools/datavalidation/. Provided that your payment form posts in utf-8 you should see that x_first_name shows up in the relay response post as 88888888.

The issue can create other problems, for example when generating xml from the field values in the relay response. The utf-8 – (n-dash  U+2013) character arrives as %0D which causes xml parsers to choke.

Even more serious probably is the issue of not reliably being able to compare data in the relay response with data coming directly from the form prior to posting to the gateway. Since the gateway may "corrupt" non-latin foreign characters arriving at the relay response URL, it becomes very difficult to detect whether any of the name or address fields were tampered with, prior to the transaction posting to the payment gateway.

The ideal solution to this would be for the relay response encoding to default to utf-8 (instead of iso-8859-1 ?) or at least provide a setting to force this encoding.

Any help with this would be appreciated since we rely on order data coming through the relay-response. If we cannot get utf-8 data through this path, we will have to re-architect our payment process to bypass the gateway to collect order data directly from browsers (and only use the gateway for payment-specific details).

thanks!

[RichardH]

Hello jldec,

Our system has some preliminary support for Unicode, but only fully supports the ISO 8859-1 block. It is likely that you will see errors when submitting other characters.

I've reported your issue to our product team for consideration in a future release.

I'd recommend subscribing to this topic so that you'll be alerted via email if there are updates. To subscribe, click Topic Options at the top of this thread and then select Subscribe. You'll then receive an email once anyone replies to your post.

Thanks,

Richard

[jldec]

That's what we expected - thanks for validating (and please do continue pushing for a fix).

We have started the process of adjusting our DPM form to post non-credit-card fields separately in an AJAX call, bypassing the gateway.

j.

[sscotti]

I think that I am having an issue with this  as well.  I am using UTF-8 on our server and for our database.  When I POST our form to you using the SIM method I am urlencoding with PHP 'Aeroméxico" to Aerom%C3%A9xico.  In the relay response, that is what is getting passed back to me, it seems, and when I urldecode it I get back Aeroméxico, but something strange is going on because when I send it out in e-mail using 8-bit, UTF-8 it doesn't display correctly and I think there is an encodin issue, or I need to tweak the mbstring settings on my server.

Are you using UTF-8 to encode and send your data, or are you still using ISO 8859-1 ?  If so, how do I deal with that when passing data containing special characters back and forth to you.  I really only have a couple of custom fields that need encoding.