It will be good if we can discuss what task you want to accomplish based on that we can suggest best practices that other partners are following to access large amounts of data. You can write your comments below.