Converting String To Android Jsonobject Loses Utf-8

I am trying to get a (JSON formatted) String from a URL and consume it as a Json object. I lose UTF-8 encoding when I convert the String to JSONObject. This is The function I use t

Solution 1:

You're telling Java to convert the string (with key message) to bytes using ISO-8859-1 and than to create a new String from these bytes, interpreted as UTF-8.

newString(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8");

You could simply use:

Stringmessages= reader.getString("messages");

Solution 2:

You can update your code as the following:

privatestatic String getUrlContents(String theUrl) {
        StringBuildercontent=newStringBuilder();
        try {
            URLurl=newURL(theUrl);
            URLConnectionurlConnection= url.openConnection();
            BufferedReaderbufferedReader=newBufferedReader(newInputStreamReader(urlConnection.getInputStream(), "utf-8"));

            String line;
            while ((line = bufferedReader.readLine()) != null) {
                content.append(line).append("\n");
            }
            bufferedReader.close();
        } catch(Exception e) {
            e.printStackTrace();
        }

        return content.toString().trim();
    }

Solution 3:

You've got two encoding issues:

1. The server sends text encoded in a character set. When you setup your InputStreamReader, you need to pass the encoding the server used so it can be decoded properly. The character encoding is usually given in the Content-type HTTP response, in the charset field. JSON is typically UTF-8 encoded, but can also be legally UTF-16 and UTF-32, so you need to check. Without a specified encoding, your system environment will be used when marshalling bytes to Strings, and vice versa . Basically, you should always specify the charset.

2. String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8"); is obviously going to cause issues (if you have non-ascii characters) - it's encoding the string to ISO-8995-1 and then trying to decode it as UTF-8.

A simple regex pattern can be used to extract the charset value from the Content-type header before reading the inputstream. I've also included a neat InputStream -> String converter.

privatestatic String getUrlContents(String theUrl) {

    try {
        URLurl=newURL(theUrl);
        URLConnectionurlConnection= url.openConnection();
        InputStreamis= urlConnection.getInputStream();

        // Get charset field from Content-Type headerStringcontentType= urlConnection.getContentType();
        // matches value in key / value pairPatternencodingPattern= Pattern.compile(".*charset\\s*=\\s*([\\w-]+).*");
        MatcherencodingMatcher= encodingPattern.matcher(contentType);
        // set charsetString to match value if charset is given, else default to UTF-8StringcharsetString= encodingMatcher.matches() ? encodingMatcher.group(1) : "UTF-8";

        // Quick way to read from InputStream.// \A is a boundary match for beginning of the inputreturnnewScanner(is, charsetString).useDelimiter("\\A").next();
    } catch(Exception e) {
        e.printStackTrace();
    }

    returnnull;
}

Solution 4:

Not sure if this will help, but you might be able to do something like this:

JSONObject result = null;
String str = null;
try 
{           
    str = newString(output, "UTF-8");
    result = (JSONObject) newJSONTokener(str).nextValue();
} 
catch (Exception e) {}

String messages = result.getString("messages");