Anders Hesselbom

Compressing a web request

2010-09-12

Perhaps you remember my old GZip example that illustrated how to use the GZip compression feature of the .NET Framework. To follow, read the Main function. It starts with a 118 character string that is compressed to a array of 208 bytes, than decompressed and displayed.

(The functions are not changed since wrote about them the last time.)

Module Module1

    Sub Main()

        'Create some text.
      Dim S As String = "This is some text that I want to compress. " & _
      "Preferably it's a long string loaded from a text file or some XML document."
      Console.WriteLine("Original string charachter count: {0}", S.Length)

        'Assign the compressed version to the variable B.
      Dim B() As Byte = CompressText(S.ToString())
      Console.WriteLine("Compressed to bytes: {0}", B.Length)

        'Decompress it, and display the result.
      Dim Decompressed As String = DecompressText(B)
      Console.WriteLine("Decompressed text:")
        Console.WriteLine(Decompressed)

      Console.WriteLine()
      Console.Write("Press Enter to quit.")
      Console.ReadLine()

    End Sub

    Private Function CompressText(ByVal T As String) As Byte()
        Dim B() As Byte = System.Text.Encoding.UTF8.GetBytes(T)
        Using MemStream As New System.IO.MemoryStream()
            Using GZStream As New System.IO.Compression.GZipStream(MemStream, _
             IO.Compression.CompressionMode.Compress)
                GZStream.Write(B, 0, B.Length)
                GZStream.Close()
                Return MemStream.ToArray()
            End Using
        End Using
    End Function

    Private Function DecompressText(ByVal B() As Byte) As String
        Dim Result As New System.Text.StringBuilder()
        Using MemStream As New System.IO.MemoryStream(B)
            Using GZStream As New System.IO.Compression.GZipStream(MemStream, _
             IO.Compression.CompressionMode.Decompress)
                Do
                    Dim Buffer(1023) As Byte 'Note that this makes 1024 bytes in VB.
                    Dim BytesRead As Integer = GZStream.Read(Buffer, 0, 1024)
                    If BytesRead > 0 Then
                        Result.Append(System.Text.Encoding.UTF8.GetString(Buffer, 0, BytesRead))
                    End If
                    If BytesRead < 1024 Then
                        Exit Do
                    End If
                Loop
                GZStream.Close()
                Return Result.ToString()
            End Using
        End Using
    End Function

End Module

You will need more data to get a positive effect of compression, and that is one thing to consider when you decide if to compress your data. Another thing to keep in mind is that compressed data isn’t searchable.

If you were to build a web browser, one elementary thing that your program must be able to do, is to make a web request via http and download the response. This ability is also built in into the framework. This function (takes an URL and returns the HTML response) does just that:

Private Function DoWebRequest(ByVal URL As String) As String
   Dim Result As String = ""
   Using Wc As New System.Net.WebClient()
      Dim S As System.IO.Stream = Wc.OpenRead(URL)
      Using R As New System.IO.StreamReader(S)
         Result = R.ReadToEnd()
      End Using
   End Using
   Return Result
End Function

Now, if I do a small change to the program, so that it downloads the response from a specific website, and compresses that, the numbers will be better. This is what the code looks like:

Sub Main()

   'Create some text.
   Dim S As String = DoWebRequest("http://www.hurhomeopatifungerar.se/")
   Console.WriteLine("Original string charachter count: {0}", S.Length)

   'Assign the compressed version to the variable B.
   Dim B() As Byte = CompressText(S.ToString())
   Console.WriteLine("Compressed to bytes: {0}", B.Length)
 
   'Decompress it, and display the result.
   Dim Decompressed As String = DecompressText(B)
   Console.WriteLine("Decompressed text:")
   Console.WriteLine(Decompressed)

   Console.WriteLine()
   Console.Write("Press Enter to quit.")
   Console.ReadLine()

End Sub

This specific website gives a 2179 characters long response, that is compressed to 1359 bytes. This is the complete code:

Module Module1

    Sub Main()

        'Create some text.
      Dim S As String = DoWebRequest("http://www.hurhomeopatifungerar.se/")
      Console.WriteLine("Original string charachter count: {0}", S.Length)

        'Assign the compressed version to the variable B.
      Dim B() As Byte = CompressText(S.ToString())
      Console.WriteLine("Compressed to bytes: {0}", B.Length)

        'Decompress it, and display the result.
      Dim Decompressed As String = DecompressText(B)
      Console.WriteLine("Decompressed text:")
        Console.WriteLine(Decompressed)

      Console.WriteLine()
      Console.Write("Press Enter to quit.")
      Console.ReadLine()

    End Sub

   Private Function DoWebRequest(ByVal URL As String) As String
      Dim Result As String = ""
      Using Wc As New System.Net.WebClient()
         Dim S As System.IO.Stream = Wc.OpenRead(URL)
         Using R As New System.IO.StreamReader(S)
            Result = R.ReadToEnd()
         End Using
      End Using
      Return Result
   End Function

    Private Function CompressText(ByVal T As String) As Byte()
        Dim B() As Byte = System.Text.Encoding.UTF8.GetBytes(T)
        Using MemStream As New System.IO.MemoryStream()
            Using GZStream As New System.IO.Compression.GZipStream(MemStream, _
             IO.Compression.CompressionMode.Compress)
                GZStream.Write(B, 0, B.Length)
                GZStream.Close()
                Return MemStream.ToArray()
            End Using
        End Using
    End Function

    Private Function DecompressText(ByVal B() As Byte) As String
        Dim Result As New System.Text.StringBuilder()
        Using MemStream As New System.IO.MemoryStream(B)
            Using GZStream As New System.IO.Compression.GZipStream(MemStream, _
             IO.Compression.CompressionMode.Decompress)
                Do
                    Dim Buffer(1023) As Byte 'Note that this makes 1024 bytes in VB.
                    Dim BytesRead As Integer = GZStream.Read(Buffer, 0, 1024)
                    If BytesRead > 0 Then
                        Result.Append(System.Text.Encoding.UTF8.GetString(Buffer, 0, BytesRead))
                    End If
                    If BytesRead < 1024 Then
                        Exit Do
                    End If
                Loop
                GZStream.Close()
                Return Result.ToString()
            End Using
        End Using
    End Function

End Module

Categories: Visual Basic 9

Tags: Streams