# How to Scrape Everything from Telegram Using Python

## Learn how to get API keys for Telegram, extract group members, and scrape Telegram group or channel comments

**Telegram** is one of the best communications apps around the world. People usually use Telegram to manage their communities and promotions.

In this tutorial, you will learn:

- How to get API keys for Telegram
- How to extract Group members.
- How to Scrape Telegram Group or Channel Comments

# How to get API keys for Telegram

Let’s get your telegram **api_id** and **api_hash** values**.** Go to [my.telegram.org](http://my.telegram.org/) you will see this are:

Log in here to **manage your apps** using Telegram API or **delete your account**. Enter your number and we will send you a confirmation code via Telegram (not SMS).

After login, you will see this picture:

Click on **API development tools** and fill in the required fields. You can choose any name for your app. For example, I chose demo1. After submitting, you will receive App **api_id** and **api_hash** like on the picture.

Please save your **api_id** and **api_hash** values somewhere. You will use these **api_id** and **api_hash** to login to Telegram API.

# How to extract Group members

**Telethon** is an [asyncio](https://docs.python.org/3/library/asyncio.html) Python 3 [MTProto](https://core.telegram.org/mtproto) library to interact with [Telegram](https://telegram.org/)’s API as a user or through a bot account. This library is meant to make it easy for you to write Python programs that can interact with Telegram.

Let’s install telethon and pandas using pip:

```js
pip install telethon
pip install pandas
```

Let’s create requirements.txt file

```js
pip freeze > requirements.txt
numpy==1.23.1
pandas==1.4.3
pyaes==1.6.1
pyasn1==0.4.8
python-dateutil==2.8.2
pytz==2022.1
rsa==4.9
six==1.16.0
Telethon==1.24.0
```

After installing **telethon** and **pandas** library. Here is the all code to get users' information from group and put them in a csv file. **Not:** You have to paste your api_id, api_hash, phone and username.

```js
import configparser
import json
import asyncio
import pandas as pd

from telethon import TelegramClient
from telethon.errors import SessionPasswordNeededError
from telethon.tl.functions.channels import GetParticipantsRequest
from telethon.tl.types import ChannelParticipantsSearch
from telethon.tl.types import (
    PeerChannel
)

# Setting configuration values
api_id = 3..1
api_hash = ''30..e3''
api_hash = str(api_hash)
phone = ''+your_phone''
username = "your_telegram_user_name"

# Create the client and connect
client = TelegramClient(username, api_id, api_hash)

async def main(phone):
    await client.start()
    print("Client Created")
    # Ensure you're authorized
    if await client.is_user_authorized() == False:
        await client.send_code_request(phone)
        try:
            await client.sign_in(phone, input(''Enter the code: ''))
        except SessionPasswordNeededError:
            await client.sign_in(password=input(''Password: ''))

me = await client.get_me()

user_input_channel = input("enter entity(telegram URL or entity id):")

if user_input_channel.isdigit():
        entity = PeerChannel(int(user_input_channel))
    else:
        entity = user_input_channel

my_channel = await client.get_entity(entity)

offset = 0
    limit = 100
    all_participants = []

while True:
        participants = await client(GetParticipantsRequest(
            my_channel, ChannelParticipantsSearch(''), offset, limit,
            hash=0
        ))
        if not participants.users:
            break
        all_participants.extend(participants.users)
        offset += len(participants.users)

all_user_details = []
    for participant in all_participants:
        all_user_details.append(
            {"user.id": participant.id,
            "first_name": participant.first_name,
            "last_name": participant.last_name,
            "username": participant.username,
            "phone": "a_"+str(participant.phone),
            "user.access_hash": participant.access_hash,
            "is_bot": participant.bot})

#with open(''user_data.json'', ''w'') as outfile:
    #    json.dump(all_user_details, outfile)

filename = user_input_channel.replace("/","_").replace":"""
    df = pd.DataFrame(all_user_details)
    df.to_csv(''withphone_''+filename+''.csv'', index=True)

with client:
    client.loop.run_until_complete(main(phone))
```

When you run the code you have to paste the group name URL or entity id. Then you will get all users' information from the group in a CSV file.

# How to Scrape Telegram Group or Channel Comments

In this section, We will scrape messages from Telegram Groups or Channels. To do this we have to install **telethon** package and only the installation telethon package is enough to do that.

Let’s install **telethon** using pip:

```js
pip install telethon
```

Let’s create requirements.txt file

```js
pip freeze > requirements.txt
pyaes==1.6.1
pyasn1==0.4.8
rsa==4.9
Telethon==1.24.0
```

Here is the simplest code:

```js
from telethon.sync import TelegramClient
import pandas as pd

api_id = 3..1
api_hash = ''30..e3''
phone = ''+9..6''
username = "H..k"

data = []
with TelegramClient(username, api_id, api_hash) as client:
    for message in client.iter_messages("https://t.me/clickhouse_en"):
        print(message.sender_id, '':'', message.text, message.date)
        data.append([message.sender_id, message.text, message.date, message.id, message.post_author, message.views, message.peer_id.channel_id ])

df = pd.DataFrame(data, columns=["message.sender_id", "message.text"," message.date", "message.id",  "message.post_author", "message.views", "message.peer_id.channel_id" ]) # creates a new dataframe
df.to_csv(''filename.csv'', encoding=''utf-8'')
```

Scraped Messages:

# Here is the code block to scrape all comments from Telegram Group or Channels. **Not:** You have to paste your api_id, api_hash, phone, and username.

```js
import configparser
import json
import asyncio
from datetime import date, datetime

from telethon import TelegramClient
from telethon.errors import SessionPasswordNeededError
from telethon.tl.functions.messages import (GetHistoryRequest)
from telethon.tl.types import (
    PeerChannel
)

# some functions to parse json date
class DateTimeEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, datetime):
            return o.isoformat()

if isinstance(o, bytes):
            return list(o)

return json.JSONEncoder.default(self, o)

# Reading Configs
config = configparser.ConfigParser()
config.read("config.ini")

# Setting configuration values
api_id = 32...31 #config[''Telegram''][''api_id'']
api_hash = ''309...823e3''  #config[''Telegram''][''api_hash'']

api_hash = str(api_hash)

phone = ''+90..76'' #config[''Telegram''][''phone'']
username = "Heyhak" #config[''Telegram''][''username'']

# Create the client and connect
client = TelegramClient(phone, api_id, api_hash)

me = await client.get_me()

user_input_channel = input(''enter entity(telegram URL or entity id):'')

if user_input_channel.isdigit():
        entity = PeerChannel(int(user_input_channel))
    else:
        entity = user_input_channel

my_channel = await client.get_entity(entity)

offset_id = 0
    limit = 100
    all_messages = []
    total_messages = 0
    total_count_limit = 0

while True:
        print("Current Offset ID is:", offset_id, "; Total Messages:", total_messages)
        history = await client(GetHistoryRequest(
            peer=my_channel,
            offset_id=offset_id,
            offset_date=None,
            add_offset=0,
            limit=limit,
            max_id=0,
            min_id=0,
            hash=0
        ))
        if not history.messages:
            break
        messages = history.messages
        for message in messages:
            all_messages.append(message.to_dict())
        offset_id = messages[len(messages) - 1].id
        total_messages = len(all_messages)
        if total_count_limit != 0 and total_messages >= total_count_limit:
            break

with open(''channel_messages.json'', ''w'') as outfile:
        json.dump(all_messages, outfile, cls=DateTimeEncoder)

with client:
    client.loop.run_until_complete(main(phone))
```

When you run the code you have to paste the group or channel name URL or entity id.

# Conclusion

Let’s recap what we have learned In this tutorial, We started off with how to get api_id and api_hash from Telegram. Then installed **telethon** package. Finally, get scraped users from the Telegram group and scraped messages from the Telegram group or channel.