There are two words for “elephant” in Latin, and both have to do with Greeks in Italy.
The first one, elephantus (or sometimes elephas or elephans), is fairly straightforward. It’s used in Latin from about the second century BC, and it’s a borrowing of the Greek word ὁ ἐλέφας, –αντος. That’s the normal Greek word for both African and Indian elephants, and is also used in Homer to mean “ivory”. By the second century BC, there were plenty of Greek-speakers in Italy, including upper-class Romans who could easily have come across the term in Herodotus, or heard it from a well-travelled friend. The spelling <ph> for Greek φ also shows that the Romans were probably preserving the aspirated pronunciation that was found in the Greek word (a p followed by an h sound, not as we say it now like an English f).
But in fact, the Romans already had a word for “elephant”, which stuck around as a poetic variant: Luca bos, or “Lucanian cow”. This term seems completely ridiculous at first sight, since of course there are no elephants native to Lucania (modern Basilicata, in Southern Italy). The idea of a “Lucanian cow” came about because of the use of war elephants in invasions of Italy.
Hannibal’s use of elephants is fairly famous, but before him came Pyrrhus of Epirus. King Pyrrhus fought a number of campaigns against the Romans in Italy in the late third century BC, supported by troops from most of the local Greek and Oscan populations. As a well-connected Hellenistic Greek monarch, he raised funds for his trip from all over the Greek East, and was given 20 (Indian) war elephants by another Hellenistic king. One of his biggest victories was the Battle of Heraclea in Lucania in 280 BC. As a result of Pyrrhus’ campaigns, Roman soldiers had seen elephants – not in north Africa, which belonged to Rome only after Hannibal’s time, but in Lucania. King Pyrrhus and his war against the Romans are somewhat forgotten, except in the phrase “Pyrrhic victory” (those elephants turned out not to be such a good idea after all) and in the Latin Luca bos.
This small example just goes to show how complicated language contact can be. The two different ways in which Latin speakers created terms for a new animal they had never seen before happened at two different points in their history. Why didn’t they borrow the Greek word the first time round? By the time the word elephantus was borrowed, were Romans more comfortable with Greek? That could make sense in terms of the history of the Roman Empire, and the expansion into the Greek East. But if you come across the term Luca bos in a poem, remember that even this apparently purely Latin term came about because of the Greeks in Italy too.