Click here to Skip to main content
15,885,435 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
This takes ages to retrieve data. Is there a more efficient way to do this?
Per_page limit is 100 & I needed the function to be able to find results for any date range using pagination. So I added a do while loop and this is what I ended up with:


What I have tried:

async function foo(repoOwner, repository, startingDate, endingDate){
    const listOfUserObjects = [];
    let pages, page = 1;
    do{
        await axios({
            method: 'get',
            url : `https://api.github.com/repos/${repoOwner}/${repository}/pulls?state=all&per_page=100&page=${page}`,
        }).then(response => {
            const users = response.data, startAt = new Date(startingDate), endAt = new Date(endingDate.valueOf());
            endAt.setDate(endAt.getDate() + 1);
            if(page === 1) pages = (Math.floor((users[0].number)/100))/2, console.log("collecting data, please wait...");
            for(let i = 0; i < users.length; i++) {
                const createdDate = new Date(users[i].created_at), updatedDate = new Date(users[i].updated_at), closedDate = new Date(users[i].closed_at), mergedDate = new Date(users[i].merged_at);
                if(((createdDate || updatedDate || closedDate || mergedDate) >= startAt) && ((createdDate || updatedDate || closedDate || mergedDate) <= endAt)) {
                    const userObj = {};
                    userObj.id = users[i].id;
                    userObj.user = users[i].user.login;
                    userObj.title = users[i].title;
                    userObj.state = users[i].state;
                    userObj.created_at = users[i].created_at.slice(0,10);
                    listOfUserObjects.push(userObj);
                } else {
                    continue;
                };
            }
            page++
        }).catch(error => {
            throw error.response.status === 404 ? Error("Error 404 User or Repo Not Found") : Error(error);
        });
    } while(page < pages);
    console.log(listOfUserObjects);
}
Posted
Comments
Member 15627495 14-Nov-22 11:34am    
1- :: you are looping on too much Date objects.
as they are coming as input, you can put all those 'new date(...)' outside the loop. so place them before all loops, at the begining of the function as firt lines.
imagine the gain * 100 each load. startingDate and endingDate will be initialized just ONE time.

2- :: another location is about the "if( || || || || ) >startat && .... <=end at " you can delete this test. because you implement the code yourself, so go 'optimistic/no failure' . it will pass. errors wont happen at this point. you know recursion when you buy a bike and you ask yourself twice "is this a bike ?". useless test in fact. if you init a var at 0, var = 0; you won't test the line after if(var == 0 )...
Lindo Matabane 15-Nov-22 6:05am    
Hello & thanks for the reply. The issue is that with the code I have I need the Date objects to filter the data. On my side if I move them out then my code no longer works as expected.
Member 15627495 15-Nov-22 5:48am    
another way, with vars and types :
don't abuse of conception, by your code : 'response.data' becomes 'users' , then 'users' becomes 'userObj'. finally you push 'userObj' in 'listOfUserObjects'.
It's 4 array vars, 4 containers for the same datas ... allocation, reallocation. instead of all that, why don't you use a global var by native JS ?
var USERS = response.datas;
by this one the 'for loop' in your code can be delete. and the array USERS contains all datas for the display( after you retrieve all answer by AJAX) .

so you have to create 1 var 'USERS' to pick a copy of 'response.data' content instead of the 'for' loop .
Lindo Matabane 15-Nov-22 6:02am    
One of my constraints is to avoid global variables that aren't modules & no use of Ajax either.
Member 15627495 15-Nov-22 10:15am    
can you copy/paste an original value of 'startingDate' please ?

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900