109. Processing of large csv files

The HttpResponse object will return the responded data as a whole, at which point if the data is very large, the browser will exceed the default timeout and return the timeout if it does not get a response from the server for a long time.Streaming HttpResponse, on the other hand, returns the responded data to the browser as a stream of data so that the browser can receive the data sent by the server without returning a timeout because the server has not responded for a long time.

StreamingHttpResponse:

This class is designed to handle streaming data so that when processing large files, connections do not time out because the server takes too long to process.This class does not inherit HttpResponse, but inherits HttpResponseBase and differs from HttpResponse in the following ways:

(1) This class does not have an attribute content, instead streaming_content.
(2) The streaming_content of this class must be an object that can be iterated over.
(3) There is no write method for this class. Writing data to objects of this class will result in an error.
Note: StreamingHttpResponse starts a process class and maintains a long connection with the client, so it is very resource intensive, so try to use this method less if it is not a special requirement.
1. Use StreamingHttpResponse to generate large csv files and download them. When accessing them in the browser, be careful to observe the loading in the lower left corner of the browser, or when making a secure connection and when downloading. The sample code is as follows:
from django.http import HttpResponse, StreamingHttpResponse
from django.template import loader


# Define a class Echo to implement the write method
class Echo:
    def write(self, value):
        return value


def large_csv(request):
# Gets a StreamingHttpResponse object and sets the file type to csv
    response = StreamingHttpResponse(content_type='text/csv')
    # Does not display the contents of the file, but can be downloaded and set the name of the file
    response['Content-Disposition'] = "attachment; filename=large_csv.csv"
# Because the first parameter passed by the StreamingHttpResponse object must be an Iterable object
    # Traversing with for... In... Gives us a generator-like object that doesn't return data all at once.
    rows = ("row {},{}\n".format(row,row) for row in range(0,10000000))
    # Use the format() function to replace the subsequent row with the {} in the string
    response.streaming_content = rows
    return response
2. Use HttpResponse to get large csv files and download them. Be sure to observe when the files in the lower left corner of the browser are loaded securely and when they are downloaded. The sample code is as follows:
from django.http import HttpResponse
import csv


def large_view(request):
    # Initialization
    response = HttpResponse(content_type='text/csv')
    # Specifies that the file does not display the contents of the file, can be downloaded as an attachment, and specifies the file name
    response['Content-Disposition'] = "attachment; filename=large_csc.csv"
    
    <!--Use python Built-in csv In module write()Method Direction response Object Write Data-->
    writer = csv.writer(response)
    for row in range(0,10000000):
        writer.writerow(['row {}'.format(row),'row'.format(row)])
    return response
131 original articles published, 30 praised and 8345 visited
Private letter follow

Tags: Django Attribute less Python

Posted on Mon, 10 Feb 2020 17:29:30 -0800 by mebar3